Multi-information Complementarity Neural Networks for Multi-Modal Action Recognition

Abstract

Multi-modal methods play an important role on action recognition. Each modal can extract different features to analyze the same motion classification. But numbers of researches always separate the one task from the others, which cause the unreasonable utilization of complementary information in the multi-modality data. Skeleton is robust to the variation of illumination, backgrounds and viewpoints, while RGB has better performance in some circumstances when there are other objects that have great effect on recognition of action, such as drinking water and eating snacks. In this paper, we propose a novel Multi-information Complementarity Neural Network (MiCNN) for human action recognition to address this problem. The proposed MiCNN can learn the features from both skeleton and RGB data to ensure the abundance of information. Besides, we design a weighted fusion block to distribute the weights reasonably, which can make each modal draw on their respective strengths. The experiments on NTU RGB-D datasets demonstrate the excellent performance of our scheme, which are superior to other methods that we have ever known.