HAN: An efficient hierarchical self-attention network for skeleton-based gesture recognition

Abstract

Previous methods for skeleton-based gesture recognition mostly arrange the skeleton sequence into a pseudo image or spatial–temporal graph and apply a deep Convolutional Neural Network (CNN) or Graph Convolutional Network (GCN) for feature extraction. Although achieving superior results, the computing efficiency still remains a serious issue. In this paper, we concentrate on designing an extremely lightweight model for skeleton-based gesture recognition using pure self-attention module. With dynamic attention weights, self-attention module is able to aggregate the features of the most informative joints using a shallow network. Considering the hierarchical structure of hand joints and inspired by the idea of divide-and-conquer, we propose an efficient hierarchical self-attention network (HAN) for skeleton-based gesture recognition. The hierarchical design can further reduce computation cost and allow the network to explicitly extract finger-level spatial temporal features, which further improves the performance of the model. Specifically, the joint self-attention module is used to capture spatial features of fingers, the finger self-attention module is designed to aggregate features of the whole hand. In terms of temporal features, the temporal self-attention module is utilized to capture the temporal dynamics of the fingers and the entire hand. Finally, these features are fused by the fusion self-attention module for gesture classification. Experiments show that our method achieves competitive results on three gesture recognition datasets with much lower computational complexity.