Low-Rank Transformer for High-Resolution Hyperspectral Computational Imaging

Abstract

Spatial-spectral fusion aims to obtain high-resolution hyperspectral image (HR-HSI) by fusing low-resolution hyperspectral image (LR-HSI) and high-resolution multispectral image (MSI). Recently, many convolutional neural network (CNN)-based methods have achieved excellent results. However, these methods only consider local contextual information, which limits the fusion performance. Although some Transformer-based methods overcome this problem, they ignore some intrinsic characteristics of HR-HSI, such as spatial low-rank characteristics, resulting in large parameters and high computational cost. To address this problem, we propose a low-rank Transformer network (LRTN) for spatial-spectral fusion. LRTN can make full use of the spatial prior of MSI and the spectral prior of LR-HSI, thereby achieving outstanding fusion performance. Specifically, in the feature extraction stage, we utilize the cross-attention mechanism to force the model to focus on spatial information that is not available in LR-HSI and spectral information that is not available in MSI. In the feature fusion stage, we carefully design a self-attention mechanism guided by spatial and spectral priors to improve spatial and spectral fidelity. Moreover, we present a novel spatial low-rank cross-attention module, which can better capture global spatial information compared to other Transformer structures. In this module, we combine the matrix factorization theorem to fully exploit the spatial low-rank characteristics of HSI, which reduces parameters and computational cost while ensuring fusion quality. Experiments on several datasets demonstrate that our method outperforms the current state-of-the-art spatial-spectral fusion methods.