UMSCS: A Novel Unpaired Multimodal Image Segmentation Method Via Cross-Modality Generative and Semi-supervised Learning

Abstract

Multimodal medical image segmentation is crucial for enhancing diagnostic accuracy in various clinical settings. However, due to the difficulty of obtaining complete data in real clinical settings, the use of unpaired and unlabeled multimodal data is severely limited. This results in unpaired data being unusable as simultaneous input for models due to spatial misalignments and morphological differences, and unlabeled data failing to provide effective supervisory signals for models. To alleviate these issues, we propose a semi-supervised multimodal segmentation method based on cross-modal generative that seamlessly integrates image translation and segmentation stages. In the cross-modalities generative stage, we employ adversarial learning to discern the latent anatomical correlations across various modalities, followed by maintaining a balance between semantic consistency and structural consistency in image translation through region-aware constraints and cross-modal structural information contrastive learning with dynamic weight adjustment. In the segmentation stage, we employ a teacher-student semi-supervised learning (SSL) framework where the student network distills multimodal knowledge from the teacher network and utilizes unlabeled source data to enhance the supervisory signal. Experimental results demonstrate that our proposed method achieves state-of-the-art performance in extensive experiments on the segmentation tasks of cardiac substructures and multi-organs abdominal, outperforming other competitive methods.