Domain disentanglement and fusion based on hyperbolic neural networks for zero-shot sketch-based image retrieval

Abstract

With the advancement of zero-shot sketch-based image retrieval (ZS-SBIR) tasks, existing methods still encounter two major challenges: Euclidean space fails to effectively represent data with hierarchical structures, leading to non-discriminative retrieval features; relying solely on visual information is insufficient to align cross-domain features and maximize their domain generalization capabilities. To tackle these issues, this paper designs a hyperbolic neural networks based ZS-SBIR framework that considers domain disentanglement and fusion learning, called “DDFUS”. Specifically, we present a contrastive cross-modal learning method that guides the alignment of multi-domain visual representations with semantic representations in the hyperbolic space. This approach ensures that each visual representation possesses rich semantic hierarchical structure information. Furthermore, we propose a domain disentanglement method based on hyperbolic neural networks that employs paired hyperbolic encoders to decompose the representation of each domain into domain-invariant and domain-specific features to reduce information disturbance between domains. Moreover, we design an advanced cross-domain fusion method that promotes the fusion and exchange of multi-domain information through the reconstruction and generation of cross-domain samples. It significantly enhances the representation and generalization capabilities of domain-invariant features. Comprehensive experiments demonstrate that the mAP@all of our DDFUS model surpasses CNN-based models by 18.99 % on the Sketchy dataset, 1.93 % on the more difficult TU-Berlin dataset, and 11.4 % on the more challenging QuickDraw dataset.