TRICAN: Multi-Modal Hateful Memes Detection with Triplet-Relation Information Cross-Attention Network

Memes are spreading on social networking. Most are created to be humorous, while some become hateful with the combination of images and words, conveying negative information to people. The hateful memes detection poses an interesting multimodal fusion problem, unlike traditional multi-modal tasks, the majority of memos have images and text that are only weakly consistent or even uncorrelated, so various modalities contained in the data play an important role in predicting its results. In this paper, we attempt to work on the Facebook Meme challenge, which solves the binary classification task of predicting a meme's hatefulness or not. We extract triplet-relation information from origin OCR text features, image content features and image caption features and proposed a novel cross-attention network to address this task. TRICAN leverages object detection and image caption models to explore visual modalities to obtain “actual captions” and then combines combine origin OCR text with the multi-modal representation to perform hateful memes detection. These meme-related features are then reconstructed and fused into one feature vector for prediction. We have performed extensively experimental on multi-modal memory datasets. Experimental results demonstrate the effectiveness of TRICAN and the usefulness of triplet-relation information.