MAUN: Memory-Augmented Deep Unfolding Network for Hyperspectral Image Reconstruction

Abstract

Spectral compressive imaging has emerged as a powerful technique to collect the 3D spectral information as 2D measurements. The algorithm for restoring the original 3D hyperspectral images (HSIs) from compressive measurements is pivotal in the imaging process. Early approaches painstakingly designed networks to directly map compressive measurements to HSIs, resulting in the lack of interpretability without exploiting the imaging priors. While some recent works have introduced the deep unfolding framework for explainable reconstruction, the performance of these methods is still limited by the weak information transmission between iterative stages. In this paper, we propose a Memory-Augmented deep Unfolding Network, termed MAUN, for explainable and accurate HSI reconstruction. Specifically, MAUN implements a novel CNN scheme to facilitate a better extrapolation step of the fast iterative shrinkage-thresholding algorithm, introducing an extra momentum incorporation step for each iteration to alleviate the information loss. Moreover, to exploit the high correlation of intermediate images from neighboring iterations, we customize a cross-stage transformer (CSFormer) as the deep denoiser to simultaneously capture self-similarity from both in-stage and cross-stage features, which is the first attempt to model the long-distance dependencies between iteration stages. Extensive experiments demonstrate that the proposed MAUN is superior to other state-of-the-art methods both visually and metrically. Our code is publicly available at https://github.com/HuQ1an/MAUN.