Image fusion: A deep Y shaped–residual convolution auto-encoder with MS-SSIM loss function

Abstract

Image fusion and deep learning are actively investigating fields of research. Their application domains include machine vision, clinical imaging, remote sensing, and other areas, all of which are used to obtain comprehensive information about a specific image. Image fusion is a process that integrates multiple imaging modalities to create a single image, for the sake of providing comprehensive information. Extensive literature shows that various methodologies, requirements, and network types are utilized for diverse modality fusion. This paper addresses the previously described issue by utilizing a unique Y-shaped Residual Convolution Autoencoder Neural Network to combine images from various modalities using the same network specifications and thereby eliminating the need for manual fusion. The combined convolved features are recreated in the decoder part using a symmetric nested residual approach with the encoder. By employing MS-SSIM as the loss function, the network is capable of generating images that are perceptually and pixel-wise indistinguishable from the target images. The fusion results are compared with five other current approaches, and the Y-shaped convolutional autoencoder result demonstrates superior achievement in both quantitative and qualitative aspects.