SFA-Net: A SAM-guided focused attention network for multimodal remote sensing image matching

Abstract

The robust and accurate matching of multimodal remote sensing images (MRSIs) is crucial for realizing the fusion of multisource remote sensing image information. Traditional matching methods fail to exhibit effective performance when confronted with significant nonlinear radiometric distortions (NRDs) and geometric differences in MRSIs. To address this critical issue, we propose a novel framework called the SAM-guided Focused Attention Network for MRSI matching (SFA-Net). Firstly, we utilize the Segment Anything Model to extract the edge structural features of MRSIs. In the meantime, convolutional neural networks are employed to extract the local deep features of MRSIs. The obtained edge structural features are then used as a prior information to guide the region self-attention network and the focused fusion cross-attention network. This improves the uniqueness of local depth features in a single image and enhances the cross-modal representation of local depth features across different images. Finally, metric learning and optimization algorithms are applied to improve the success rate of feature matching, further enhancing the accuracy and robustness of the matching results. Experimental results on 1050 MRSI pairs confirm that SFA-Net is able to achieve high-quality matching on large-scale challenging MRSI datasets, with good adaptation to severe NRDs and geometric differences. SFA-Net outperforms state-of-the-art algorithms qualitatively and quantitatively, including RIFT, ASS, CoFSM, WSSF, HOWP, CMM-Net, R2D2, ECOTR, and LightGlue. Our code¹ and dataset will be made publicly available upon publication of the paper.