PatchOut: A novel patch-free approach based on a transformer-CNN hybrid framework for fine-grained land-cover classification on large-scale airborne hyperspectral images

Abstract

Airborne hyperspectral systems can provide high-resolution hyperspectral images (HSIs) covering large scenes, enabling fine-grained land-cover classification. However, the most popular patch-based methods are limited by low computational efficiency and broken classification results, which hinders the full utilization of this powerful technology in Earth observation applications. Therefore, in this paper, considering the efficiency requirements for large-scale land-cover classification, a novel patch-free approach based on a Transformer-CNN hybrid (PatchOut) framework is proposed. The proposed PatchOut framework adopts an encoder-decoder architecture, enabling rapid semantic segmentation for HSI classification. For the encoder module, we introduce a computationally efficient reduced Transformer module integrated with convolutional neural network (CNN), to leverage their complementary strengths for long-range and local feature extraction, respectively. A multi-scale spatial-spectral feature fusion (MSSSFF) module is also proposed to amalgamate the characteristics of different levels from the encoder, which enhances the overall feature representation. Then, to address the loss of semantic detail and resolution inherent in multi-level feature extraction, a novel feature reconstruction module (FRM) is applied to recover high-quality semantic features. Finally, a large-scale benchmark dataset, Qingpu-HSI, is presented, comprising airborne HSIs covering 33.91 km² with 20 land-cover classes. Experiments on the Qingpu-HSI and another public dataset demonstrate the superior accuracy and efficiency of our proposed PatchOut framework, outperforming several well-known patch-free and patch-based methods. The Qingpu HSI dataset, along with the PatchOut framework code will be released at https://github.com/busbyjrj/PatchOut.