DSF2-NAS: Dual-Stage Feature Fusion via Network Architecture Search for Classification of Multimodal Remote Sensing Images

Abstract

Recently, the classification of multimodal remote sensing images (RSIs) has garnered significant attention due to its ability to provide rich information for various scenes on Earth. Compared to traditional feature fusion methods used for the classification of multimodal RSIs, neural architecture search (NAS) is capable of identifying the optimal network structure for multimodal RSIs and downstream tasks. However, due to the diverse spatial resolutions, complex channel dimensions, and drastic foreground scale variations of multimodal RSIs, challenges arise when employing NAS methods for precise classification: 1) Due to the complementary and redundant nature between different modalities in RSIs, determining the features within each modality for fusion becomes quite challenging; 2) the design of fusion operators does not take into account the spatial positions and channel relationships between different modalities of RSIs, making it difficult for the fused features to match downstream tasks. To address these issues, we propose a dual-stage feature fusion framework based on NAS, termed DSF2-NAS, for the classification of multimodal RSIs. It primarily consists of two components: the feature candidate search (FCS) module and the fusion operator search (FOS) module, which execute sequentially. In the FCS module, a feature distance-based regularization approach is proposed to ensure fusion using multimodal features with the highest complementarity. Meanwhile, in the FOS module, a series of fusion operators are designed, which are based on spatial positions, channel relationships, and self-attention mechanisms, aiming to better integrate multimodal features with complex spatial and channel information. The proposed method has been evaluated on various datasets of multimodal RSIs, and experimental results consistently show that this method achieves state-of-the-art performance across multiple classification metrics.