A Hybrid Convolutional and Graph Neural Network for Human Action Detection in Static Images

Abstract

Human action detection in static images is a hot and challenging field within computer vision. Given the limited features of a single image, achieving precision detection results require the full utilization of the image’s intrinsic features, as well as the integration of methods from other fields to process the images for generating additional features. In this paper, we propose a novel dual pathway model for action detection, whose main pathway employs a convolutional neural network to extract image features and predict the probability of the image belonging to each respective action. Meanwhile, the auxiliary pathway uses a pose estimate algorithm to obtain human key points and connection information for constructing a graphical human model for each image. These graphical models are then transformed into graph data and input into a graph neural network for features extracting and probability prediction. Finally, a corresponding connected neural network propose by us is used to fusing the probability vectors generated from the two pathways, which learns the weight of each action class in each vector to enable their subsequent fusion. It is noted that transfer learning is also used in our model to improve the training speed and detection accuracy of it. Experimental results upon three challenging datasets: Stanford40, PPMI and MPII illustrate the superiority of the proposed method.