SiMultiF: A Remote Sensing Multimodal Semantic Segmentation Network With Adaptive Allocation of Modal Weights for Siamese Structures in Multiscene

Abstract

Semantic segmentation of remote sensing images is crucial for resource exploration, precision agriculture, and environmental monitoring. However, conducting semantic segmentation on single-modality data for remote sensing images that contain various scenes, especially unique scenes, is highly challenging. To address this challenge, we propose SiMultiF, a Siamese architecture-based multimodal feature adaptive fusion semantic segmentation network. SiMultiF employs a dual-branch Siamese structure feature extractor. The adaptive feature weight adjustment module (AFWAM) and the multimodal fusion module (MFM) facilitate in-depth understanding and extraction of multimodal data. Specifically, the Siamese structure can extract features from multimodal data concurrently without adding to the number of parameters. The AFWAM module can adaptively identify the importance of different modal data and dynamically adjust the modal weight to enhance the network’s comprehension of complex scene data. Additionally, the cross-attention (CA)-based MFM module bridges modality gaps and achieves comprehensive multimodal feature fusion. Numerous experiments have demonstrated that the proposed SiMultiF outperforms other state-of-the-art semantic segmentation models (both multimodal and single modal) on the high-resolution ISPRS Potsdam dataset, ISPRS Vaihingen dataset, and special scene dataset (vegetation polarization dataset with extreme natural lighting contrast). Moreover, the robustness and generalizability of the network in multiscene and multimodal datasets are verified.