SR_ColorNet: Multi-path attention aggregated and mask enhanced network for the super resolution and colorization of panchromatic image

Abstract

Improving the spatial and spectral resolution of remote sensing images is important in areas such as environmental monitoring and military reconnaissance. High-quality remote sensing images with high spatial and spectral resolution often yield better results in these fields. However, a single sensor cannot obtain high spatial resolution color remote sensing images and can only obtain grayscale panchromatic (PAN) images and low spatial resolution multispectral(MS) images. In existing methods, obtaining high spatial resolution color images is tough when only PAN images are input. Image super-resolution (SR) models can improve the spatial resolution of the image, but not the spectral resolution. Image colorization models can improve spectral resolution, not spatial resolution. Pan-sharpening models depend on paired PAN and MS images. This study aggregates SR and colorization tasks of PAN images in the same model and completed simultaneously. We propose a multi-path network (SR_ColorNet) for recovering PAN image resolution, utilizing both Transformer and Convolutional Neural Network (CNN) architectures. Our method includes three key stages: shallow feature extraction, deep feature extraction, and feature reconstruction. Shallow feature extraction employs a VGG19 for multi-path feature extraction. The deep feature extraction stage consists of three modules: the SR transformer (SRT) module for recovering spatial information, ECA Channel Mixing Block (ECMB) for retaining and transmitting significant feature information, and Fusion Feature Processing Block (FFPB) for processing information. In the feature reconstruction stage, a Masked Feature Enhancement (MFE) module is proposed to enhance the feature. Our SR_ColorNet performed well at image SR and colorization in experiments, according to objective metrics and visual quality. Our code is available at https://github.com/QianqianWang1325/SR_ColorNet_main.