Spatio-Temporal 3D Action Recognition with Hierarchical Self-Attention Mechanism

Abstract

3D action recognition is a long-standing problem in the field of computer vision. Given the 3D coordinate set of body joints, it is desired to recognize what activity is performed. The problem can be approached using a time-series model. Recent advancements in the field of recurrent neural networks have enabled the use of sophisticated memory cells that can predict time series using the information from earlier elements of a sequence. In this article, we proposed a hierarchical architecture that attends to its own signature through time, which can put more weight on time frames of the sequence that are more specific to the performed action. Accordingly, using memory cells, a self-attention mechanism is implemented. In addition, spatial attention is also considered by sub-grouping and then regrouping body parts down the architecture hierarchy. We evaluate the proposed model on NTU and MSR 3D action datasets. An accuracy of 79.8% and 97.8% on NTU and MSR datasets indicated that the proposed method outperforms the previous methods tested in this paper.