Multi-scale Graph Convolutional Network for understanding human action in videos

Abstract

Temporal action detection aims to classify and locate human action in videos, which has been a difficult challenge in the field of smart transportation and intelligent manufacturing. In general, an action consists of multiple action units, and understanding the relationships between different action units is beneficial for detection. Most existing methods simply use the temporal context and ignore the relationships between the action units. In this paper, we propose a Multi-scale Graph Convolutional Network (MGCN), which can capture the different relationships between action units. Specifically, MGCN includes a Graph Pyramid Module (GPM), which divides a video into different scales and builds a graph for each scale. In each graph, we model action units as nodes and their relationships as edges. In addition, we propose GPM-T, a generalized version of GPM that can be plugged into state-of-the-art methods (e.g., G-TAD, TemporalMaxer) to enhance their performance. Experimental results show that at tIoU 0.5, MGCN reaches 72.8%, 33.8%, 20.2% and 17.9% mAP on THUMOS14, MultiTHUMOS, EPIC-Kitchens 100 ($verb$ and $noun$), surpassing existing state-of-the-art methods. In addition, GPM-T improves the mAP of G-TAD and TemporalMaxer from 42.2% and 71.8% to 46.2% and 72.1%, respectively. The source code can be found at https://github.com/mugenggeng/GPM-T.git.