Image–text sentiment analysis based on hierarchical interaction fusion and contrast learning enhanced

Abstract

As social media continues to evolve, an increasing number of individuals prefer to publish a variety of content that combines different forms of expression on platforms to convey their feelings. In recent times, the study of sentiment within these diverse media formats has gained significant traction. However, some studies have neglected the multilayered interactions between modalities and underutilized the relationships between data samples and between classes. To address this problem, this paper proposes an image–text sentiment analysis method based on layer interaction fusion and contrast learning enhanced. First, the method uses a multi-layer cross-modal interaction module to emphasize the complementarity between modalities. The correlation information between modalities is deeply mined. Then, feature integration is performed by the multimodal fusion module. In addition, the model introduces a comparative learning task to exploit the relationship between samples and between classes. The emotional features under different classes and the key emotional features of the samples are mined. Finally, sentiment classification is performed to realize image–text sentiment analysis. In order to assess the efficacy of the approach, this paper performs extensive experimental validation on a multimodal sentiment dataset. Compared with several baseline models, the model in this paper achieves certain improvements. For example, on the multi-view sentiment analysis single dataset(MVSA-Single dataset), the F1 score is improved by 1.90% compared to the baseline model.