Hand Gesture Recognition with ConvNets for School-Aged Children to Learn Basic Arithmetic Operations

Abstract

Hand Gesture and Sign Language Recognition are recognized as non-trivial problems in the computer vision community. Deep Learning (DL) has provided both, a novel analytical framework and solutions. Moreover, Convolutional Neural Networks (CNN) have shown there are still goals to be met and solutions to be given regarding Hand Posture Recognition with the architectures and technologies we have today. This work proposes building a Hand Gesture Recognizer able to identify 13 classes: numbers from 0 to 9, and the signs corresponding to the arithmetic operations of addition, subtraction, and multiplication. The core model aims to translate digits and math symbols signed by students and teachers in a school environment to both make the teaching-learning process more engaging and to promote sign language learning in the student community. Based on Object Detection and leveraging a pretrained model using Transfer Learning, this DL model was retrained with a data set of 1,365 images (105 per class/sign) of Panamanian Sign Language hand-shapes. An accuracy of 88% was reached on the validation data, with proved usefulness on American Sign Language (ASL) for these hand-shapes, and being easily adaptable to other sign systems, including two-handed finger spelling ones like International Sign System (IS).