Interpretation of Sri Lankan Sign Language: A Wearable Sensor-based Approach

Abstract

Hearing-impaired and speech-impaired people communicate not only with themselves but also with ordinary people using visual languages. Sri Lankan Sign Language (SSL) is the standard visual language used in Sri Lanka. Like other sign languages, the SSL relies on a distinct combination of hand gestures, body movements, and facial expressions for communication. As a result, SSL is more challenging for individuals without knowledge of SSL to understand. On the other hand, the steep learning curve associated with SSL makes it even more difficult to acquire. Thus, the interpretation of SSL has become a need. However, Sri Lanka is suffering from a severe dearth in the availability of SSL interpreters. This justifies the need to use either vision-based or sensor-based technological approaches to help the interpretation of SSL. However, vision-based approaches are susceptible to conditions such as skin tone, background color, ambient light intensity, and real-time constraints, whilst the sensor-based solutions are generally better in gesture recognition. Further, there is no attempt has been made on developing a cost-effective, portable, and real-time solution to accurately interpret the hand gestures of SSL. In this paper, we, therefore, present a novel, wearable, sensor-based, real-time gesture recognition glove, and a machine-learning Long Short-Term Memory (LSTM) model to recognize the hand and finger positions in three-dimensional space for classification and interpretation of SSL. The proposed approach has achieved 320ms of lowest inference time while showing a promising result of 83% for categorical accuracy. Our aim is to help the interpretation of SSL with an affordable, portable as well as a real-time solution.