Understanding Social Relations with Graph-Based and Global Attention

Abstract

Social relations, as the basic relationships in our daily life, are a phenomenon unique to human society that shows how people interact in society. Social relations understanding is to infer the existing social relationships between individuals in a given scenario, which is crucial for us to analyze social behavior. Existing research methods are usually limited to extracting features of characters and related entities, which limits the scope of attention and may miss important clues such as interactions between characters. In this paper, we propose a global attention mechanism that adaptively grasps scenes, objects, and human interactions for reasoning about social relationships. We propose an end-to-end global attention network, which consists of three modules, namely, a convolutional attention module, a graph inference module, and an attentional inference module. The visual and location information is first extracted by the convolutional attention module as the feature information of the person pairs, then it is made to process the relationships between character nodes on the graph inference network, and finally, the attention is fully utilized to classify the social relationships. Extensive experiments on the PISC and PIPA datasets show that our proposed method outperforms the state-of-the-art methods in terms of accuracy.