INF3: Implicit Neural Feature Fusion Function for Multispectral and Hyperspectral Image Fusion

Abstract

Multispectral and Hyperspectral Image Fusion (MHIF) is a task that aims to fuse a high-resolution multispectral image (HR-MSI) and a low-resolution hyperspectral image (LR-HSI) acquired on the same scene to obtain a high-resolution hyperspectral image (HR-HSI). Benefiting from the powerful inductive bias capability, convolutional neural network (CNN) based methods have achieved great success for the MHIF task. However, they lack flexibility when processing multi-scale images and require convolution structures be stacked to enhance performance. Implicit neural representation (INR) has recently achieved good performance and interpretability in 2D processing tasks thanks to its ability to locally interpolate samples and utilize multimodal content, such as pixels and coordinates. Although INR-based approaches show promising results, they put additional demands on high-frequency information (e.g., positional encoding). In this paper, we propose the use of the HR-MSI as high-frequency detail auxiliary input, thus introducing a new INR-based hyperspectral fusion function called implicit neural feature fusion function (INF ³ ). The method overcomes the inherent shortcomings of vanilla INR thereby solving the MHIF problem. Specifically, our INF ³ designs a dual high-frequency fusion (DHFF) structure that obtains high-frequency information from HR-MSI and LR-HSI fusing them with coordinate information. Moreover, the proposed INF ³ incorporates a parameter-free method called INR with cosine similarity (INR-CS) that uses cosine similarity to generate local weights through feature vectors. Relied upon INF ³ , we build an implicit neural fusion network (INFN) that achieves state-of-the-art performance for the MHIF task on two public datasets, i.e., CAVE and Harvard. It also reaches the advanced level on the Pansharpening task, proving the flexibility of the proposed approach.