Textual semantics enhancement adversarial hashing for cross-modal retrieval

Abstract

Supervised cross-modal hashing seeks to embed rich semantic information into binary hash codes, thereby enhancing semantic discrimination. Despite substantial advancements in cross-modal semantic learning, two critical challenges remain: (1) the fine-grained semantic information inherent in individual words within text contents is underutilized; and (2) more efficient constraints are required to mitigate the distributional heterogeneity across modalities. To overcome these issues, we introduce a Textual Semantics Enhancement Adersarial Hashing method, abbreviated as TSEAH, aimed at further improving hashing retrieval performance. Our approach introduces an effective textual semantics enhancement strategy involving a Bag-of-Words Self-Attention (BWSA) mechanism, which accentuates fine-grained semantics from textual content. This mechanism facilitates the transfer of fine-grained semantic knowledge from texts to images. Furthermore, we incorporate an adversarial hashing strategy within the cross-modal hashing learning process to ensure semantic distribution consistency across different modalities. Importantly, our solution achieves impressive results without the need for complex visual-language pre-training models. Comparative evaluations across three commonly used datasets demonstrate that our method achieves outstanding average accuracy: 90.41$\text{\%}$ on MIRFLICKR-25K, 82.86$\text{\%}$ on NUW-SIDE, and 83.53$\text{\%}$ on MS COCO, outperforming the state-of-the-art baselines by a significant margin ranging from 1.97$\text{\%}$ to 2.51$\text{\%}$.