FakeCLIP: Multimodal Fake Caption Detection with Mixed Languages for Explainable Visualization

Abstract

Existing fake news research relies on news propagation or news metadata. Waiting for propagation structure to be enough is a waste of time. Hoping for reliable metadata information is also a waste because all data can be forged. The most natural way for human when verifying a news is through the content itself. In social media, most of the circulating news are in minimal content which consist of image and its text caption. We propose FakeCLIP to examine whether a caption truly describes the corresponding image or not. As far as we know, we are the first one to tackle fake news using fake caption approach. We found mixed languages problem where one single text can consist of many different languages mixed together. We provide explainable visualization for intuitive reasoning of which part contains fake information. Moreover, we also consider alignment of what happens in the image that being discussed in the text caption while showing the fake signal over them. Our proposed method performs better than the current state-of-the-art on Twitter datasets by 11.1%.