Resolving multimodal ambiguity via knowledge-injection and ambiguity learning for multimodal sentiment analysis

Abstract

Multimodal Sentiment Analysis (MSA) utilizes complementary multimodal features to predict sentiment polarity, which mainly involves language, vision, and audio modalities. Existing multimodal fusion methods primarily consider the complementarity of different modalities, while neglecting the ambiguity caused by conflicts between modalities (i.e. the text modality predicts positive sentiment while the visual modality predicts negative sentiment). To well diminish these conflicts, we develop a novel multimodal ambiguity learning framework, namely RMA, Resolving Multimodal Ambiguity via Knowledge-Injection and Ambiguity Learning for Multimodal Sentiment Analysis. Specifically, We introduce and filter external knowledge to enhance the consistency of cross-modal sentiment polarity prediction. Immediately, we explicitly measure ambiguity and dynamically adjust the impact between the subordinate modalities and the dominant modality to simultaneously consider the complementarity and conflicts of multiple modalities during multimodal fusion. Experiments demonstrate the dominantity of our proposed model across three public multimodal sentiment analysis datasets CMU-MOSI, CMU-MOSEI, and MELD.