Cyclic Cross-Modality Interaction for Hyperspectral and Multispectral Image Fusion

Abstract

Integrating low-resolution hyperspectral images with high-resolution multispectral images is an effective approach to derive high-resolution hyperspectral images. Recently, numerous deep learning-based approaches have been employed to model the mapping relationships for the fusion directly. However, these methods often neglect the spectral characteristics and fail to facilitate comprehensive interactions among global features from heterogeneous modalities. In this paper, we propose a novel cyclic Transformer based on the cross-modality spatial-spectral interaction, exploiting diverse interaction modes to explore the similarity and complementarity among cross-modality features. Specifically, we design a cyclic interactive architecture to fully exploit the abundant spectral prior information in low-resolution hyperspectral images and the rich spatial prior information in high-resolution multispectral images. By incorporating spatial and spectral priors into the attention mechanisms in Transformer modules, we explore the long-range dependency information within the cross-modality features. Furthermore, to enhance interaction among features from different modalities, we devise the cross-modality adaptive interaction mechanisms in both spatial and spectral dimensions to facilitate information reciprocity between different modalities. Extensive experiments demonstrate that the proposed approach outperforms the state-of-the-art fusion methods both quantitatively and visually. The code is available at https://github.com/Tomchenshi/CYformer.