Recognition of Human-object Interaction in Video through a Two-stream Network Integrating Multiple Features

To understand a scene, a machine not only has to learn to recognize individual object instances, but also pair them up and recognize visual relationships between them. Human-object interaction (HOI) recognition is one of the fundamental tasks in understanding the visual world. In this paper, we address the task of understanding and recognizing HOI in videos, and represent HOI as the doublet of . This paper proposes a two-stream network model that fuses multiple features. Since appearance information (the appearance features of instances), spatial information (the distance between each key part of the human body and the interacting object), and motion information (optical flow) in the video are all essential cues for recognizing HOI, our model uses two streams to fuse the information to complete the HOI recognition task. We valid the effectiveness of the model by conducting experiments on two recently proposed public video datasets (Charades and CAD-120 datasets), and perform ablation experiments to show the effect of components.