German Sign Language Translation using 3D Hand Pose Estimation and Deep Learning

Abstract

Sign language is the primary medium of communication for the majority of the world’s population suffering from disabling hearing loss that creates a barrier between the hearing and the hearing-impaired people. In this paper, sign language translation is undertaken for German Sign Language (GSL) characters from a single image by leveraging the technique of 3D object detection. We make use of a three-network architecture that performs segmentation, keypoint localization, and elevation from a two-dimensional plane to the three-dimensional space, from a single RGB image containing the signed gesture. Thirty gestures have been used and the best results were obtained using a combination of pose representation coordinates, joint angles, and pool layer features of AlexNet for classification. The system gives a character error rate of 0.29, a reduction of error rate by 12.12% when compared to the state-of-the-art approach.