Cross-modal evidential fusion network for social media classification

Abstract

Human activities on social networks can reflect the attitudes of the masses towards various events and are important for economic development and social progress. Many studies have focused on various multimodal tasks in social media, driven by the development of deep multimodal techniques. However, existing multimodal methods treat both reliable and unreliable modalities equally, which affects the efficiency of multimodal classification underlying social media. Therefore, a reliable method for multimodal fusion is required. This study presents a novel cross-modal evidential fusion network (CEFN) based on the subjective logic theory to incorporate uncertainty estimates into the multimodal fusion process. CEFN models uncertainty directly and learns more reliable representations by treating the outputs of encoders as subjective opinions. To reduce semantic uncertainty caused by random noise, momentum models are employed for each unimodal encoder. These unimodal encoders align the pseudo-views generated by the momentum models to mitigate the effects of noise. In addition, CEFN introduces a conflict loss function to facilitate representation learning from image-text pairs containing opposing views. This loss captures uncertainty from cross-modal conflicts to improve the feature extraction capability of each encoder. Experimental results on three real-world social media datasets show that CEFN outperforms related multimodal networks.