Nighttime visible and infrared image fusion based on adversarial learning

Abstract

The task of infrared–visible image fusion (IVIF) aims to integrate multi-modal complementary information and facilitate other downstream tasks, especially under some harsh circumstances. To tackle the challenges of preserving significant information and enhancing visual effects under nighttime conditions, we propose a novel IVIF method based on adversarial learning, namely AdvFusion. It consists of an autoencoder-based generator and a dual discriminator. In particular, the multi-scale features of source images are firstly extracted by ResNet, and then aggregated based on the attention mechanisms and nest connection strategy to generate the fused images. Meanwhile, a global and local dual discriminator structure is designed to minimize the distance between the illumination distributions of the reference images and fused images, which achieves contrast enhancement within fused images and helps to uncover hidden cues in darkness. Moreover, a color loss is utilized to maintain color balance of each fused image, while the widely used perceptual loss and gradient loss are employed to maintain content consistency between the source and fused images. Extensive experiments conducted on five datasets demonstrate that our AdvFusion can achieve promising results compared with the state-of-the-art IVIF methods in terms of both visual effects and quantitative metrics. Furthermore, AdvFusion can also boost the performance of semantic segmentation on MSRS dataset and object detection on M3FD dataset.