LEFuse: Joint low-light enhancement and image fusion for nighttime infrared and visible images

Abstract

Infrared and visible image fusion (IVIF) aims to represent scenes more richly and accurately by integrating information from both modalities. However, existing IVIF methods are typically designed for normal illumination conditions, aiming to achieve higher scores by maintaining close similarity to the source images. In night scenes, visible images often suffer from both low light and localized overexposure due to the dim environment and the interference from local light sources. These methods fail to explore the information hidden in the dark regions of visible images, resulting in fusion images that lack texture details, appear overall dark, and exhibit poor visual quality. To address this issue, we propose a novel image fusion network called LEFuse. LEFuse not only integrates complementary information from both visible and infrared images but also focuses on recovering hidden texture details in visible images. By doing so, LEFuse enhances the visibility and contrast of the fused image, resulting in a brighter and more vivid representation. To achieve this goal, we propose a set of unsupervised loss functions to drive the network’s learning. This set includes a maximum entropy-based fusion enhancement loss for both image fusion and low-light enhancement, as well as a perceptual loss to mitigate the impact of local overexposure in visible images on the fused result. These losses can be applied to any existing image fusion network, enhancing fused images without compromising fusion performance. Extensive experiments demonstrate that our LEFuse achieves promising results in terms of visual quality and quantitative evaluations, especially in nighttime environments. Our code is publicly available at https://github.com/cheng411523/LEFuse.