Fine-Tuning Foundation Models With Confidence Assessment for Enhanced Semantic Segmentation

Abstract

Confidence assessments of semantic segmentation algorithms are important. Ideally, models should have the ability to predict in advance whether their output is likely to be incorrect. Assessing the confidence levels of model predictions in Earth observation (EO) classification is essential, as it can enhance semantic segmentation performance and help prevent further exploitation of the results in the case of erroneous prediction. The model we developed, Confidence Assessment for enhanced Semantic segmentation (CAS), evaluates confidence at both the segment and pixel levels, providing both labels and confidence scores as output. Our model, CAS, identifies segments with incorrectly predicted labels using the proposed combined confidence metric, refines the model, and enhances its performance. This work has significant applications, particularly in evaluating EO Foundation Models on semantic segmentation downstream tasks, such as land-cover classification using Sentinel-2 satellite data. The evaluation results show that this strategy is effective and that the proposed model CAS outperforms other baseline models.