Finger movements tracking of the human hand using the smart camera to control the Allegro Hand robot

Abstract

Gestures are one of the richest ways of communication that humans have. Therefore, automated hand detection and finger behavior are some of the most compelling challenges in robot vision. Recently, many ideas and technologies have been used in programming to use cameras to recognize the fingers of human hands and apply them to robots. In this paper; the innovative algorithm is based on intelligent ideas and mainly relies on image analysis after camera shooting. In order to identify the fingers of the hand by a camera, the idea is to assign each finger to a percentage in pixels of the binary image, which represents the area of the finger. Each ratio compares to the live image to determine which fingers the user has chosen to control the Allegro Hand robot. Also in this work, the hand can move freely in front of the camera, relying on a new idea of measuring the distance between the hand and the camera. We use for this a small ticket in geometric form square and color different to black and white. This ticket is affixed to the back of the user's hand and through which we can know the size of the hand, regardless of the distance on the Z-axis of the image. This is important in determining the proportions of the fingers according to the distance from the camera. The camera is here combined with a hand human and Allegro Hand to ensure more interactive communication, harmony and easier. The result of the smart camera is tested with multiple experimental trials with a robotic hand. Many finger positions were identified correctly in front of the camera.