TempTrans-MIL: A General Approach to Enhancing Multimodal Tactile-Driven Robotic Manipulation Classification Tasks

Abstract

In tactile-driven robotic manipulation, handling high-dimensional, multimodal tactile time series data from different tactile sensors presents challenges in feature extraction, processing, and interpretation. This study tackles these challenges by proposing a robust method that effectively processes tactile time series data, without visual input. We propose the temporal transformer-based multiple instance learning (TempTrans-MIL) model, a deep learning approach for high-dimensional tactile time series data in robotic manipulation classification tasks. The model is an MIL-based framework treats each short-term multimodal tactile time series as a bag of instances. It uses an inception module-based encoder for instance-level temporal feature extraction, and an MIL module to integrate bag-level features using tokenized transformers with learnable wavelet positional encoding. Extensive experiments in robotic manipulation tasks, using both publicly available and our own collected dataset, demonstrate that our proposed TempTrans-MIL model significantly outperforms baselines. Our proposed model achieves a good balance between classification accuracy and computational efficiency, with accuracies of 88.50% for surface material recognition, 91.75% for grasp stability detection, and 99.31% for robotic palpation task, highlighting its superior performance across various tasks.