Hybrid Joint Embedding with Intra-Modality Loss for Image-Text Matching

Abstract

Image-text(caption) matching has become a regular evaluation of joint-embedding models that combine vision and language. This task comprises ranking the data of one modality (images) based on a Text query (Image Retrieval), and ranking texts by relevance for an image query (Text Retrieval). The current joint embedding approaches use symmetric similarity measurement, due to that order embedding is not taken in consideration. In addition to that, in image-text matching, the used losses ignore the intra similarity in a certain modality that explores the relation between the candidates in the same modality. In spite of, the important role of intra information in the embedding. In this paper, we proposed a hybrid joint embedding approach that combines between distance preserving which based on symmetric distance and order preserving that based on asymmetric distance to improve image-text matching. In addition to that we propose an intra loss function to enrich the embedding with intra-modality information. We evaluate our embedding approach on the baseline model on Flickr30K dataset. The proposed loss shows a significant enhancement in matching task.