EnsCLR: Unsupervised skeleton-based action recognition via ensemble contrastive learning of representation

Skeleton-based action recognition is a key research area in video understanding, beneficial from its compact and efficient motion information. To relieve from the burden of expensive and laborious data annotation, unsupervised approaches, particularly contrastive learning, have been widely employed to extract action representations from unlabeled data. In this paper, we propose an Ensemble framework for Contrastive Learning of Representation (EnsCLR) to preform unsupervised skeleton-based action recognition. Concretely, Queue Extension method is devised to generate discriminative representation by aggregating the ensemble information from multiple pipelines. Furtherly, Ensemble Nearest Neighbors Mining (ENNM) method is utilized to excavate the most similar samples from the unlabeled data as positive samples, which alleviates the false-negative samples problem caused by the disregard of category label. The experiments with extensive evaluation protocols show that EnsCLR outperforms previous state-of-the-art methods on NTU60, NTU120, and PKU-MMD datasets.