TCIP: Network with topology capture and incongruity perception for sarcasm detection

Abstract

Multimodal sarcasm detection is a pivotal visual-linguistic task that aims to identify incongruity between the text purpose and the underlying meaning of other modal data. Existing works are dedicated to the learning of unimodal embeddings and the fusion of multimodal information. Nonetheless, they neglect the importance of topology and incongruity between multimodal information for sarcasm detection. Therefore, we propose a novel multimodal sarcasm detection network that incorporates multimodal topology capture and incongruity perception (TCIP). A text single-mode graph, a visual single-mode graph, and a visual–text heterogeneous graph are first established, where nodes contain visual elements and text elements. The association matrix of the heterogeneous graph encapsulates visual–visual associations, text–text associations, and visual–text associations. Subsequently, TCIP learns single-modal graphs and a heterogeneous graph based on graph convolutional networks to capture text topology information, visual topology information, and multimodal topology information. Furthermore, we pull together multimodal embeddings exhibiting consistent distributions and push away those with inconsistent distributions. TCIP finally feeds the fused embedding into a classifier to detect sarcasm results within visual–text pairs. Experimental results conducted on the multimodal sarcasm detection benchmarks and the multimodal science question answering dataset demonstrate the exceptional performance of TCIP.