4DCov: A Nested Covariance Descriptor of Spatio-Temporal Features for Gesture Recognition in Depth Sequences

Abstract

In this paper we propose a novel covariance-based framework for the robust characterization and classification of human gestures in 3D depth sequences. The proposed 4DCov descriptor uses the notion of covariance to create compact representations of complex interactions between variations of 3D features in the spatial and temporal domain, instead of using the absolute features themselves. Despite the compactness of this representation, it still offers discriminative power for human-gesture classification. The codification of feature variations along a scene makes our descriptor robust to inter-subject and intra-class variations, periodic motions and different speeds during gesture executions, compared to other key point or histogram-based descriptor approaches. Furthermore, a sparse collaborative classification method is also presented, taking advantage of our descriptor laying on a specific manifold topology and observing that similar motions are geometrically clustered in the descriptor space. Classification accuracy results are presented against state-of-the-art approaches on top of four public human gesture datasets acquired with 3D depth sensor devices, including complex gestures from different natures.