Attribute-Image Similarity Measure for Multimodal Attention Mechanism

Abstract

Multimodal attention mechanisms in computer vision applications enable rich feature extraction by attending to specific image regions, highlighted through a second mode of data regarded as auxiliary information. The correspondence between image regions and auxiliary data can be defined as the similarity between parts of the two modes. In this paper, we propose a similarity measure that maximizes the posterior for matching high-level object attributes with image regions. In contrast to previous methods, we rely on attribute space rather than textual descriptions. We evaluate our results on the CUB dataset. The results show that the proposed method better minimizes the similarity loss function compared to the text-image similarity measurement.