Time Series Neural Networks for Real Time Sign Language Translation

Abstract

Sign language is the primary mode of communication for the hearing and speech impaired and there is a need for systems to translate sign languages to spoken languages. Prior research has been focused on providing glove based solutions which are intrusive and expensive. We propose a sign language translation system based solely on visual cues and deep learning for accurate translation. Our system applies Computer Vision and Neural Machine Translation for American Sign Language (ASL) gloss recognition and translation respectively. In this paper, we show that an end to end neural network system is not only capable of recognition of individual ASL glosses but also translation of continuous sign language videos into complete English sentences, making it an effective and practical tool for sign language communication.