Intrinsically stretchable sensory-neuromorphic system for sign language translation

Abstract

Soft wearable strain sensors with mechanically invisible interactions with skin tissue have enabled precise diagnosis and effective treatment of neurological movement disorders in a closed-loop manner that quantitatively measures motion-related strains without noise intervention and provides feedback information. Because of the immediate interpretation from motion-driven sign language to general conversation, such on-skin strain sensors have recently been considered promising candidates for facilitating communication either within deaf and hard-of-hearing communities or among people with disabilities. Despite advances in soft strain sensors, the lack of intrinsically stretchable neuromorphic modules that mimic biological synapses and efficiently perform neural computation and dynamics has resulted in inaccurate translation of sign language. In this study, we present an intrinsically stretchable organic electrochemical transistor (is-OECT) synapse integrated with crack-based strain sensors conformally mounted onto fingers to implement an interactive sensory-neuromorphic system (iSNS) capable of overcoming auditory impediments. The is-OECT synapse in the iSNS shows stable electrical performance (a large number of states (∼100 states) and a linear weight update) in the skin deformation range (approximately 30%). Based on pre-trained data gathered from on-finger strain-sensing information, the iSNS wirelessly translates sign language while maintaining high accuracy.