STPGANsFusion: Structure and Texture Preserving Generative Adversarial Networks for Multi-modal Medical Image Fusion

Abstract

Medical images from various modalities carry diverse information. The features from these source images are combined into a single image, constituting more information content, beneficial for subsequent medical applications. Recently, deep learning (DL) based networks have demonstrated the ability to produce promising fusion results by integrating the feature extraction and preservation task with less manual interventions. However, using a single network for extracting features from multi-modal source images characterizing distinct information results in the loss of crucial diagnostic information. Addressing this problem, we present structure and texture preserving generative adversarial networks based medical image fusion method (STPGANsFusion). Initially, the textural and structural components of the source images are separated using structure gradient and texture decorrelating regularizer (SGTDR) based image decomposition for more complementary information preservation and higher robustness for the model. Next, the fusion of the structure and the texture components is carried out using two generative adversarial networks (GANs) consisting of a generator and two discriminators to get fused structure and texture components. The loss function for each GAN is framed as per the characteristic of the component being fused to minimize the loss of complementary information. The fused image is reconstructed and undergoes adaptive mask-based structure enhancement to further boost its contrast and visualization. Substantial experimentation is carried out on a wide variety of neurological images. Visual and qualitative results exhibit notable improvement in the fusion performance of the proposed method in comparison to the state-of-the-art fusion methods.