AMMUNet: Multiscale Attention Map Merging for Remote Sensing Image Segmentation

Abstract

The advancement of deep learning has driven notable progress in remote sensing semantic segmentation. Multihead self-attention (MSA) mechanisms have been widely adopted in semantic segmentation tasks. Network architectures exemplified by Vision Transformers have implemented window-based operations in the spatial domain to reduce computational costs. However, this approach comes at the expense of a weakened capacity to capture long-range dependencies, potentially limiting their efficacy in remote sensing image processing. In this letter, we propose AMMUNet, a UNet-based framework that employs multiscale attention map (AM) merging, comprising two key innovations: the attention map merging mechanism (AMMM) module and the granular multihead self-attention (GMSA). AMMM effectively combines multiscale AMs into a unified representation using a fixed mask template, enabling the modeling of a global attention mechanism. By integrating precomputed AMs in preceding layers, AMMM reduces computational costs while preserving global correlations. The proposed GMSA efficiently acquires global information while substantially mitigating computational costs in contrast to the global MSA mechanism. This is accomplished through the strategic alignment of granularity and the reduction of relative position bias parameters, thereby optimizing computational efficiency. Experimental evaluations highlight the superior performance of our approach, achieving remarkable mean intersection over union (mIoU) scores of 75.48% on the challenging Vaihingen dataset and an exceptional 77.90% on the Potsdam dataset, demonstrating the superiority of our method in precise remote sensing semantic segmentation. Codes are available at https://github.com/interpretty/AMMUNet .