A multimodal embedding transfer approach for consistent and selective learning processes in cross-modal retrieval

Abstract

Cross-modal retrieval (CMR) aims to retrieve semantically relevant samples based on the query of a different modality. Previous work usually employs pair-wise and class-wise learning to learn a shared embedding space, so that modality invariance and semantic discrimination can be preserved. However, pair-wise and class-wise learning are conventionally considered separately in previous methods, which often brings about the inconsistent combination of learning objectives and unselective optimization of multimodal pairs, leading to insufficient/ineffective information utilization that degrades model performance. To tackle these issues, in this paper, we propose a novel multimodal embedding transfer approach to enable consistent and selective learning processes for CMR. To support consistent combination and maximize information utilization, our proposed framework leverages a source embedding model generated by class-wise learning and a target embedding model generated by pair-wise learning. We then develop the embedding transfer strategy to transfer multimodal embeddings from the source model to the target model, which provides the relaxed margins and relaxed labels simultaneously for the selective optimization of multimodal pairs. We finally design a soft contrastive loss to realize the multimodal embedding transfer strategy. Extensive experiments on the benchmark multimodal datasets verify the effectiveness of our approach for cross-modal retrieval.