Hybrid Pixel-Wise Registration Learning for Robust Fusion-Based Hyperspectral Image Super-Resolution

Abstract

Hyperspectral image (HSI) super-resolution (SR) aims to generate a high resolution (HR) HSI in both spectral and spatial domains, in which the fusion-based SR methods have shown great potential in producing a pleasing HR HSI by taking both advantages of the observed low-resolution (LR) HSI and HR multispectral image (MSI). Most existing fusion-based methods implicitly assume that the observed LR HSI and HR MSI are exactly registered, which is, however, difficult to comply with in practice and thus impedes their generalization performance in real applications. To mitigate this problem, we propose a hybrid pixel-wise registration learning framework for fusion-based HSI SR, which shows two aspects of advantages. On the one hand, a pixel-wise registration module (PRM) is developed to directly estimate the transformed coordinate of each pixel, which enables coping with various complex (e.g., rigid or nonrigid) misalignment between two observed images and is pluggable to any other existing architectures. On the other hand, a hybrid learning scheme is conducted to jointly learn both the PRM and the deep image prior-based SR network. Through compositing supervised and unsupervised learning in a two-stage manner, the proposed method is able to exploit both the image-agnostic and image-specific characteristics to robustly cope with unknown misalignment and thus improve the generalization capacity. Experimental results on four benchmark datasets show the superior performance of the proposed method in handling fusion-based HSI SR with various unknown misalignments.