Guanqun Cao , Jiaqi Jiang , Danushka Bollegala , Min Li , Shan Luo
{"title":"Multimodal zero-shot learning for tactile texture recognition","authors":"Guanqun Cao , Jiaqi Jiang , Danushka Bollegala , Min Li , Shan Luo","doi":"10.1016/j.robot.2024.104688","DOIUrl":null,"url":null,"abstract":"<div><p>Tactile sensing plays an irreplaceable role in robotic material recognition. It enables robots to distinguish material properties such as their local geometry and textures, especially for materials like textiles. However, most tactile recognition methods can only classify known materials that have been touched and trained with tactile data, yet cannot classify unknown materials that are not trained with tactile data. To solve this problem, we propose a tactile Zero-Shot Learning framework to recognise materials when they are touched for the first time, using their visual and semantic information, without requiring tactile training samples. The biggest challenge in tactile Zero-Shot Learning is to recognise disjoint classes between training and test materials, i.e., the test materials that are not among the training ones. To bridge this gap, the visual modality, providing tactile cues from sight, and semantic attributes, giving high-level characteristics, are combined together and act as a link to expose the model to these disjoint classes. Specifically, a generative model is learnt to synthesise tactile features according to corresponding visual images and semantic embeddings, and then a classifier can be trained using the synthesised tactile features for zero-shot recognition. Extensive experiments demonstrate that our proposed multimodal generative model can achieve a high recognition accuracy of 83.06% in classifying materials that were not touched before. The robotic experiment demo and the FabricVST dataset are available at <span>https://sites.google.com/view/multimodalzsl</span><svg><path></path></svg>.</p></div>","PeriodicalId":49592,"journal":{"name":"Robotics and Autonomous Systems","volume":null,"pages":null},"PeriodicalIF":4.3000,"publicationDate":"2024-03-24","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.sciencedirect.com/science/article/pii/S092188902400071X/pdfft?md5=644cdaa618ea404cb0bb3be3f39edd0c&pid=1-s2.0-S092188902400071X-main.pdf","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Robotics and Autonomous Systems","FirstCategoryId":"94","ListUrlMain":"https://www.sciencedirect.com/science/article/pii/S092188902400071X","RegionNum":2,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"AUTOMATION & CONTROL SYSTEMS","Score":null,"Total":0}
引用次数: 0
Abstract
Tactile sensing plays an irreplaceable role in robotic material recognition. It enables robots to distinguish material properties such as their local geometry and textures, especially for materials like textiles. However, most tactile recognition methods can only classify known materials that have been touched and trained with tactile data, yet cannot classify unknown materials that are not trained with tactile data. To solve this problem, we propose a tactile Zero-Shot Learning framework to recognise materials when they are touched for the first time, using their visual and semantic information, without requiring tactile training samples. The biggest challenge in tactile Zero-Shot Learning is to recognise disjoint classes between training and test materials, i.e., the test materials that are not among the training ones. To bridge this gap, the visual modality, providing tactile cues from sight, and semantic attributes, giving high-level characteristics, are combined together and act as a link to expose the model to these disjoint classes. Specifically, a generative model is learnt to synthesise tactile features according to corresponding visual images and semantic embeddings, and then a classifier can be trained using the synthesised tactile features for zero-shot recognition. Extensive experiments demonstrate that our proposed multimodal generative model can achieve a high recognition accuracy of 83.06% in classifying materials that were not touched before. The robotic experiment demo and the FabricVST dataset are available at https://sites.google.com/view/multimodalzsl.
期刊介绍:
Robotics and Autonomous Systems will carry articles describing fundamental developments in the field of robotics, with special emphasis on autonomous systems. An important goal of this journal is to extend the state of the art in both symbolic and sensory based robot control and learning in the context of autonomous systems.
Robotics and Autonomous Systems will carry articles on the theoretical, computational and experimental aspects of autonomous systems, or modules of such systems.