DS$^{2}$PN: A Two-Stage Direction-Aware Spectral-Spatial Perceptual Network for Hyperspectral Image Reconstruction

Abstract

Coded aperture snapshot spectral imaging (CASSI) systems are designed to modulate and compress 3D hyperspectral images (HSIs) into 2D measurements, which can capture HSIs in dynamic scenes. How to faithfully recover 3D HSIs from 2D measurements becomes one of the challenges. Impressive results have been achieved by deep leaning methods based on convolutional neural networks and transformers, but the directional information is not thoroughly explored to reconstruct HSIs and evaluate the reconstruction quality. In view of this, we propose a two-stage direction-aware spectral-spatial perceptual network (DS $^{2}$ PN) for HSI reconstruction. In the first stage, we design a frequency-based preliminary reconstruction subnetwork to roughly recover the global spectral-spatial information of HSIs via frequency interactions. In the second stage, we design a multi-directional spectral-spatial refinement subnetwork to recover the details of HSIs via directional attention mechanisms. To train the whole network, we build a pixel-level reconstruction loss for each subnetwork, and a feature-level multi-directional spectral-spatial perceptual loss which is specially tailored to high-dimensional HSIs. Experimental results show that our DS $^{2}$ PN outperforms state-of-the-art methods in quantitative and qualitative evaluation for both simulation and real HSI reconstruction tasks.