Global and local semantic enhancement of samples for cross-modal hashing

Abstract

Hashing becomes popular in cross-modal retrieval due to its exceptional performance in both search and storage. However, existing cross-modal hashing (CMH) methods may (a) neglect to learn sufficient modal-specific information, and (b) fail to fully exploit sample semantics. To address these issues, we propose a method called Semantic Enhancement of Sample Hashing (SESH). First, SESH employs a global modal-specific learning strategy to draw overall shared information and global modal-specific information by factoring the mapping matrix. Second, SESH introduces manifold learning and a local modal-specific learning strategy to extract additional local modal-specific and modal-shared data under label guidance. Meanwhile, local modal-specific information is integrated with global modal-specific details to add rich modal-specific information. Third, SESH uses discrete maximum similarity and orthogonal constraint transformation to enhance both global and local semantic information, embedding more discriminative information into the Hamming space. Finally, an efficient discrete optimization algorithm is proposed to generate the hash codes directly. Experiments on three datasets demonstrate the superior performance of SESH. The source code will be available at https://github.com/kokorording/SESH.