Utilizing Gramian Angular Fields and Convolution Neural Networks in Flex Sensors Glove for Human–Computer Interaction

Abstract

The current sensor systems using the human–computer interface to develop a hand gesture recognition system remain challenging. This research presents the development of hand gesture recognition with 16-DoF glove sensors combined with a convolution neural network. The flex sensors are attached to 16 pivot joints of the human hand on the glove so that each knuckle flex can be measured while holding the object. The 16-DoF point sensors collecting circuit and adjustable buffer circuit were developed in this research to work with the Arduino Nano microcontroller to record each sensor's signal. This article investigates the time-series data of the flex sensor signal into 2-D colored images, concatenating the signals into one bigger image with a Gramian angular field and then recognition through a deep convolutional neural network (DCNN). The 16-DoF glove sensors were proposed for testing with three experiments using 8 models of DCNN recognition. These were conducted on 20 hand gesture recognition, 12 hand sign recognition, and object manipulation according to shape. The experimental results indicated that the best performance for the hand grasp experiment is 99.49% with Resnet 101, the hand sign experiment is 100% with Alexnet, and the object attribute experiment is 99.77% with InceptionNet V3.