Joint feature fusion hashing for cross-modal retrieval

Abstract

Cross-modal hashing retrieval maps data from different modalities into a common low-dimensional hash code space, enabling fast and efficient retrieval. Recently, there has been a growing interest in the cross-modal hashing retrieval approach. Nonetheless, a significant number of current methodologies overlook the influence of semantically rich features on retrieval performance. In addition, class attribute embedding is often forgotten in cross-modal feature fusion, which is crucial for learning more discriminative hash codes. To meet these challenges, we put forward a novel method, namely joint feature fusion hashing (JFFH) for cross-modal retrieval. Specifically, we use the fast language image pre-training model as the feature coding module of cross-modal data. To more effectively mitigate semantic disparities between modalities, we introduce a multimodal contrastive learning loss to strengthen the interaction between modalities and improve the semantic representation of modalities. In addition, we extract class attribute features as class embedding and integrate them with cross-modal features to enhance the semantic relationship within the fused features. To better capture both inter-modal and intra-modal dependencies as well as semantic relevance, we integrate the self-attention mechanism into the multi-modal fusion transformer encoder to facilitate efficient feature fusion. Besides, we apply label-wise high-level semantic similarity and feature-wise low-level semantic similarity to enhance the discrimination of hash codes. Our JFFH method shows better retrieval performance in large-scale cross-modal retrieval.