Transformer-based material recognition via short-time contact sensing

Abstract

Embodied intelligence needs haptic sensing for spontaneous and accurate material recognition. The haptic sensing module of an intelligent system can acquire material data through either sliding or tapping motions. Sliding movements are commonly adopted for collecting the spatial frequency features of the material but are less time-efficient than tapping. Here, we introduce a haptic sensing framework that can extract material features from short-time tapping signals. To improve the performance of material recognition, transfer learning is used by transferring the knowledge of pretrained model training on large-scale images into haptic sensing. The waveforms of the tapping signals are encoded as images to be input into a transformer model tailored for image recognition tasks. The encoding employs line graph image-point scaling, effectively accommodating signals that exhibit large variations in magnitude and temporal structures. Using the LMT haptic material database containing sliding and tapping data, our study showcases the efficacy of the proposed framework in material recognition tasks, especially for short-time ($\le$  60 ms) sensing via tapping interactions. The findings provide fresh insights into haptic sensing technologies and may help improve the physical interaction capabilities of embodied intelligence, such as medical and rescue robots.