Enhancing video rumor detection through multimodal deep feature fusion with time-sync comments

Abstract

Rumors in videos have a stronger propagation compared to traditional text or image rumors. Most current studies on video rumor detection often rely on combining user and video modal information while neglecting the internal multimodal aspects of the video and the relationship between user comments and local segment of the video. To address this problem, we propose a method called Time-Sync Comment Enhanced Multimodal Deep Feature Fusion Model (TSC-MDFFM). It introduces time-sync comments to enhance the propagation structure of videos on social networks, supplementing missing contextual or additional information in videos. Time-sync comments focus on expressing users' views on specific points in time in the video, which helps to obtain more valuable segments from videos with high density information. The time interval from one keyframe to the next in a video is defined as a local segment. We thoroughly described this segment using time-sync comments, video keyframes, and video subtitle texts. The local segment sequences are ordered based on the video timeline and assigned time information, then fused to create the local feature representation of the video. Subsequently, we fused the text features, video motion features, and visual features of video comments at the feature level to represent the global features of the video. This feature not only captures the overall propagation trend of video content, but also provides a deep understanding of the overall features of the video. Finally, we will integrate local and global features for video rumor classification, to combine the local and global information of the video. We created a dataset called TSC-VRD, which includes time-sync comments and encompasses all visible information in videos. Extensive experimental results have shown superior performance of our proposed model compared to existing methods on the TSC-VRD dataset.