A Selective Semantic Transformer for Spectral Super-Resolution of Multispectral Imagery

Abstract

Spectral super-resolution (SSR) is an important research area. It amounts at increasing the spectral resolution of a multispectral image (MSI) with a few spectral bands to obtain a hyperspectral image (HSI) with hundreds of narrow spectral bands. State-of-the-art SSR methods typically use the transformer (or its variants) to learn the spectral mapping from the MSI to the HSI. However, these methods tend to suffer from the interference of dissimilar structures due to the constraints imposed by patch-level operations. Besides, model interpretability is attributed to prior information (from data preprocessing) rather than from an end-to-end a priori learning paradigm. To address these limitations, we propose a new selective semantic transformer (SST) for SSR. Our newly developed approach first characterizes contextual semantics within homogeneous regions and realizes information interaction from heterogeneous regions. Specifically, a superpixel-based spectral learning (SSL) strategy is designed to take into account excitated-transformer spatial and spectral semantic learning, including intra- and intersuperpixel relations, as well as superpixel edge details. Moreover, multiscale and dense residual connection mechanisms are employed to model SSL modules into an end-to-end interpretable deep network for SSR. We first conducted experiments using three well-known airborne and satellite-based datasets and then evaluated the SSR performance of our method using satellite data collected from Sentinel-2 (MSI) and GF-5 (HSI) satellites. Our results demonstrate that the newly proposed SST outperforms state-of-the-art SSR methods.