SDCluster: A clustering based self-supervised pre-training method for semantic segmentation of remote sensing images

Abstract

Reducing the reliance of remote sensing semantic segmentation models on labeled training data is essential for practical model deployment. Self-supervised pre-training methods, which learn representations from unlabeled data by designing pretext tasks, provide an approach to address this requirement. One inconvenience of the currently contrastive learning-based and masked image modeling-based self-supervised methods is the difficulty in evaluating the quality of the pre-trained model without fine-tuning for semantic segmentation task. Hence, this paper proposes a pixel-level clustering-based self-supervised learning method, named SDCluster, which allows for a qualitative evaluation of the pre-trained model through visualizing the clustering results. Specifically, SDCluster extends the self-distillation framework to the pixel-level by incorporating the clustering assignment module. Then, clustering constraint modules, including prototype constraint module and semantic consistency constraint module, are designed to eliminate ineffective cluster prototypes and preserve the semantic information of ground objects. Benefiting from the correlation between pixel-level clustering and per-pixel classification of semantic segmentation, experimental results indicate that SDCluster exhibits competitive fine-tuning accuracy and robust few-shot segmentation capabilities when compared to prevalent self-supervised methods. Large-scale pre-training experiment and practical application experiment also prove the generalization ability and extensibility of the proposed method. The code and the dataset for practical application experiment are available at https://github.com/openrsgis/SDCluster.