A multi-sensor fusion method for static Chinese sign language recognition using DE–XGBoost

Abstract

Static Chinese Sign Language Recognition (SCSLR) is an important field of research in human–computer interaction and assistive technology. Traditional SCSLR methods usually rely on computer vison sensors, which are susceptible to effects such as hand shapes, lighting conditions, and occlusions, resulting in low recognition accuracy. Additionally, sensor-based SCSLR methods cannot achieve high recognition accuracy due to limited hand gesture information. In this paper, we propose a multi-sensor fusion method, using a DE–XGBoost model, to fuse the information of hand gesture and finger curvature to achieve the SCSLR, which can overcome the recognition error problems caused by insufficient sign language information. In addition, we design and implement a prototype system, which consists of a smartphone and a smart glove, to evaluate our proposed method in comparison with support vector machine (SVM), XGBoost, gcForest, and artificial neural network (ANN). Experimental results show that our proposed method achieves a better performance in terms of accuracy, robustness, and real-time processing.