A unified framework for multi-modal rumor detection via multi-level dynamic interaction with evolving stances

Abstract

With the escalating dissemination of textual and visual content on the Internet, multi-modal rumor detection has garnered significant scholarly attention in recent research studies. Currently, the prevailing methods in multi-modal rumor detection tend to emphasize the information integration from source posts and images, overlooking the dynamic interaction between multi-modal sources and evolving conversational structures. Furthermore, they fail to recognize the potential advantage that introducing evolving user stances as a form of collective decision-making can improve the model’s performance in rumor classification. In this paper, we propose a novel Evolving Stance-aware Dynamic Graph Fusion Network (ESDGFN) to address the above issues. This network aims to integrate the source, the image and the dynamic conversation graph into a unified framework. Specifically, we begin by leveraging a cross-modal transformer for fine-grained feature fusion of the multi-modal sources. Simultaneously, based on the temporal attributes of posts, we construct a set of dynamically changing conversation graphs for each conversation thread, simulating and encoding the evolving stances of users towards the target event within these conversation graphs. Subsequently, we design a multi-level fusion strategy, incorporating both coarse-grained multi-modal feature guidance and fine-grained cross-modal similarity-aware fusion. This strategy aims to generate interactively enhanced multi-modal encoding and dynamic graph representations. The experimental results on both PHEME and Twitter datasets highlight the excellence of our ESDGFN model. It achieves 90.6% accuracy on PHEME, a 3.3% improvement compared to the state-of-the-art method, and 87% accuracy on Twitter, with a 2.4% improvement.