HF2TNet: A Hierarchical Fusion Two-Stage Training Network for Infrared and Visible Image Fusion

Abstract

In the field of infrared and visible image fusion, current algorithms often focus on complex feature extraction and sophisticated fusion mechanisms, ignoring the issues of information redundancy and feature imbalance. These limit effective information aggregation. To address these issues, this paper proposes a hierarchical fusion strategy with a two-stage training network, abbreviated as HF2TNet, which achieves effective information aggregation in a staged manner. In the initial training stage, a three-stream encoder-decoder architecture is proposed, seamlessly integrating CNN and transformer modules. This architecture extracts both global and local features from visible and infrared images, capturing their shared attributes before the fusion process. Moreover, a multi-shared attention module (MSAM) is proposed to profoundly reconstruct and augment the visible and infrared features, ensuring the preservation and enhancement of details across modalities. In the subsequent stage, HF2TNet utilizes the pre-integrated features as query inputs for the dual MSAMs. These modules interact with the previously reconstructed infrared and visible features to enhance complementary information and ensure a balanced feature fusion. Experimental results indicate HF2TNet's superior performance on standard datasets like MSRS and TNO, especially in complex scenes, demonstrating its potential in multimodal image fusion.