vCapTouch: Interactive Touch Sensing Data Synthesis for Hand Gesture Recognition Based on Digital Twin

Abstract

Touch sensing is a prominent pillar technique in various human-computer interactive scenarios, especially when touchscreen-based capacitive touch sensing has become a representative in user-end electronics. An intelligent touch-sensing system captures the capacitive touch-sensing images to recognize the objects via machine learning techniques. However, collecting the training dataset is usually laborious and time-consuming, requiring specific coding skills and knowledge. In this article, we introduced vCapTouch, a data generation method to synthesize the touch sensing data, which can be directly employed to train a machine learning model and recognize the real touching behavior, significantly lowering the need for real dataset collection. The presented method is primarily based on the idea of the digital twin. We implemented the method with Unity3D, a game engine that enables high interactivity, is easy to use, and has a low cost. We evaluated the proposed method on eight users with different touch screen devices and proved the feasibility of synthesizing the touch sensing data.