Cross-Modal Attention Network for Detecting Multimodal Misinformation From Multiple Platforms

Abstract

Misinformation detection in short videos on social media has become a pressing issue due to its popularity. However, datasets for misinformation detection are limited in terms of modality and sources, hindering the development of effective detection methods. In this article, we introduce a novel dataset denoted the multiplatform multimodal misinformation (3M) dataset. Our dataset is collected specifically to investigate and address misinformation in a multimodal context. A total of 17 352 videos were collected from two prominent social media platforms, namely TikTok and Weibo. The 3M dataset covers 30 different topics, such as sports, health, news, and art, providing a diverse range of content for analysis. We propose a novel approach named cross-modal attention misinformation detection (CAMD) for effectively detecting and addressing multimodal misinformation. CAMD leverages the cross-modal attention module to facilitate effective information exchange and fusion between modalities by learning the correlations and weights among them. The cross-modal attention module is capable of learning multilevel modality correlations, focuses primarily on the interaction between multimodal sequences across different time steps, and simultaneously adjusts the information from the source modality based on the information of the target modality. Extensive experiments on the 3M dataset show that the proposed method achieves state-of-the-art performance. Specifically, CAMD achieves accuracy, F1-score, precision, and recall values of 76.86%, 58.05%, 87.86%, and 58.70%, respectively, on the 3M dataset.