Beyond-Skeleton: Zero-shot Skeleton Action Recognition enhanced by supplementary RGB visual information

Abstract

Zero-shot action recognition (ZSAR) recognizes action categories that have not appeared during the training process and has garnered widespread attention due to its potential to save costs in retraining and data annotation. We observed that the existing ZSAR method based on skeleton sequences only uses human posture information in the skeleton sequence, lacks discriminative semantic representation in some similar behavior recognition, and lacks effective interaction between different modalities, resulting in unsatisfactory performance and limited applications of the ZSAR. To solve these problems, we propose a novel method, called Beyond-Skeleton zero-shot Learning (BSZSL), which is used to enhance zero-shot Skeleton Action Recognition. Firstly, a multi-prompt learning strategy is introduced. It utilizes prompt information to guide the model to simultaneously learn complementary semantic information related to behavior categories from both skeleton sequences and RGB information, making the visual feature information more expressive. Specifically, it employs a pre-trained multimodal model to extract prior knowledge related to behaviors from RGB and then guides the skeleton sequence features using this knowledge. This enhances the complementary features of both RGB and skeleton modalities. Secondly, to constrain the mapping relationship of different modal feature information, a Contrastive Clustering (CC) module is designed. This module emphasizes the similarity of features within the same category while increasing the differences in feature mapping between different categories. Finally, evaluating our method on the NTU-60 and NTU-120 datasets with multi-split settings, the result demonstrates that our method achieves state-of-the-art performance in both zero-shot learning (ZSL) and generalized zero-shot learning (GZSL) settings.