Saccade inspired Attentive Visual Patch Transformer for image sentiment analysis

Abstract

The generation of image-evoked emotion is usually regarded as a transient process in the image sentiment analysis. However, according to the saccade mechanism of the human visual system, the evoked emotion generated during the saccade process changes over time and attention. Based on above analysis, we propose an Attentive Visual Patch Transformer (AVPT), using visual attention sequence to represent the sentiment context of images and predict the possible distribution of sentiment. In AVPT, the spatial structure in the form of patches are reconstructed and reorganized by visual attention shift sequentially. Simultaneously, the temporal characteristics of attention shift are introduced to the relative position encoding, and merged in a self-attention manner to form a spatial–temporal process similarly to the human visual system. Specifically, we propose a sequence attention shift module to simulate the saccade process, which obtains sequence attention and reduces the computational effort by group attentive convolutional gate recurrent unit. Then, a spatial–temporal correlation encoder module is proposed to encode temporal attention with spatial visual features and obtain the sequential visual features of saccade. Finally, a self-attention fusion module is used to extract the correlation hidden in the relative encoding features. Our proposed AVPT achieves excellent performance on visual sentiment distribution prediction and is comparable to state-of-the-art methods, as demonstrated by extensive experiments on the Flickr_LDL and Twitter_LDL datasets.