Adaptive Semi-automatic Robot Control by Tongue in a Remote Setting for Individuals with Tetraplegia

Individuals with tetraplegia are severely limited in terms of independence, which can lead to depression and premature death. The field of robotics has sought to create solutions for increased independence by enabling individuals with tetraplegia to perform physical activities. This includes semiautomatic solutions. The use case of an assistive robotic manipulator (ARM) and thus increased independence could be expanded by allowing remote control, e.g. when the user is lying in bed. This study presents and evaluates semi-autonomous intraoral tongue-based control of a seven degree of freedom ARM/gripper in a near and remote setting. The system consists of a tongue control interface (iTongue), a semi-automatic framework based on YOLOv5 and adaptive levels of automation, an Intel Realsense D435i camera and a JACO robotic manipulator. Two study participants completed ten rounds of controlling the JACO ARM to reach and pick up a soda bottle and pour from it in a cup. The semi-automation improved the grasping performance for both of the study participants, particularly when controlling in a remote setting, decreasing the cognitive load and the overall task completion time. The presented system has the potential to increase the independence and quality of life for individuals with tetraplegia, by enabling the user to perform physical activities even when lying in bed. Future work should include expanding the semi-automation to other activities of daily living and evaluating the system in a greater population and by individuals with tetraplegia.