Ultra-Wideband Radar-Based Two-Stream ConvNeXt-AFF Neural Network for Sign Language Gesture Recognition

Abstract

In recent years, the miniaturization of ultra-wideband (UWB) radar devices has provided new opportunities for an increasing number of research applications. In the field of human gesture recognition, UWB radar has attracted significant research interest due to its unique advantages of high measurement accuracy, strong interference rejection capabilities, and insensitivity to ambient illumination conditions during target detection. However, the current research relies on extracting gesture features from single-domain information, which does not fully utilize all the information available in the echo. To address this, we have developed a novel multidomain feature fusion model called two-stream ConvNeXt-AFF to improve the accuracy of sign language gesture identification. In our proposed model, multiscale features are first independently extracted from time-frequency spectrograms and range-Doppler signatures using dual convolutional neural network (CNN) streams. A attention feature fusion (AFF) module is then applied to integrate multimodal representations. To validate the model’s efficacy, an experimental dataset containing 2000 samples of ten different sign language gestures was collected. Results demonstrate that the two-stream ConvNeXt-AFF network can effectively recognize sign language gestures with an accuracy of 99.86%, outperforming other traditional methods.