Pub Date : 2024-10-30DOI: 10.1109/LRA.2024.3488400
L. Nowakowski;R. V. Patel
Accurately estimating tool-tissue interaction forces during robotics-assisted minimally invasive surgery is an important aspect of enabling haptics-based teleoperation. By collecting data regarding the state of a robot in a variety of configurations, neural networks can be trained to predict this interaction force. This paper extends existing work in this domain based on collecting one of the largest known ground truth force datasets for stationary as well as moving phantoms that replicate tissue motions found in clinical procedures. Existing methods, and a new transformer-based architecture, are evaluated to demonstrate the domain gap between stationary and moving phantom tissue data and the impact that data scaling has on each architecture's ability to generalize the force estimation task. It was found that temporal networks were more sensitive to the moving domain than single-sample Feed Forward Networks (FFNs) that were trained on stationary tissue data. However, the transformer approach results in the lowest Root Mean Square Error (RMSE) when evaluating networks trained on examples of both stationary and moving phantom tissue samples. The results demonstrate the domain gap between stationary and moving surgical environments and the effectiveness of scaling datasets for increased accuracy of interaction force prediction.
{"title":"Learning Based Estimation of Tool-Tissue Interaction Forces for Stationary and Moving Environments","authors":"L. Nowakowski;R. V. Patel","doi":"10.1109/LRA.2024.3488400","DOIUrl":"https://doi.org/10.1109/LRA.2024.3488400","url":null,"abstract":"Accurately estimating tool-tissue interaction forces during robotics-assisted minimally invasive surgery is an important aspect of enabling haptics-based teleoperation. By collecting data regarding the state of a robot in a variety of configurations, neural networks can be trained to predict this interaction force. This paper extends existing work in this domain based on collecting one of the largest known ground truth force datasets for stationary as well as moving phantoms that replicate tissue motions found in clinical procedures. Existing methods, and a new transformer-based architecture, are evaluated to demonstrate the domain gap between stationary and moving phantom tissue data and the impact that data scaling has on each architecture's ability to generalize the force estimation task. It was found that temporal networks were more sensitive to the moving domain than single-sample Feed Forward Networks (FFNs) that were trained on stationary tissue data. However, the transformer approach results in the lowest Root Mean Square Error (RMSE) when evaluating networks trained on examples of both stationary and moving phantom tissue samples. The results demonstrate the domain gap between stationary and moving surgical environments and the effectiveness of scaling datasets for increased accuracy of interaction force prediction.","PeriodicalId":13241,"journal":{"name":"IEEE Robotics and Automation Letters","volume":"9 12","pages":"11266-11273"},"PeriodicalIF":4.6,"publicationDate":"2024-10-30","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142598640","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-10-29DOI: 10.1109/LRA.2024.3487490
Grzegorz Bartyzel
Transferability, along with sample efficiency, is a critical factor for a reinforcement learning (RL) agent's successful application in real-world contact-rich manipulation tasks, such as product assembly. For instance, in the case of the industrial insertion task on high-mix, low-volume (HMLV) production lines, transferability could eliminate the need for machine retooling, thus reducing production line downtimes. In our work, we introduce a method called Multimodal Variational DeepMDP (MVDeepMDP) that demonstrates the ability to generalize to various environmental variations not encountered during training. The key feature of our approach involves learning a multimodal latent dynamic representation. We demonstrate the effectiveness of our method in the context of an electronic parts insertion task, which is challenging for RL agents due to the diverse physical properties of the non-standardized components, as well as simple 3D printed blocks insertion. Furthermore, we evaluate the transferability of MVDeepMDP and analyze the impact of the balancing mechanism of the generalized Product-of-Experts (gPoE), which is used to combine observable modalities. Finally, we explore the influence of separately processing state modalities of different physical quantities, such as pose and 6D force/torque (F/T) data.
{"title":"Multimodal Variational DeepMDP: An Efficient Approach for Industrial Assembly in High-Mix, Low-Volume Production","authors":"Grzegorz Bartyzel","doi":"10.1109/LRA.2024.3487490","DOIUrl":"https://doi.org/10.1109/LRA.2024.3487490","url":null,"abstract":"Transferability, along with sample efficiency, is a critical factor for a reinforcement learning (RL) agent's successful application in real-world contact-rich manipulation tasks, such as product assembly. For instance, in the case of the industrial insertion task on high-mix, low-volume (HMLV) production lines, transferability could eliminate the need for machine retooling, thus reducing production line downtimes. In our work, we introduce a method called Multimodal Variational DeepMDP (MVDeepMDP) that demonstrates the ability to generalize to various environmental variations not encountered during training. The key feature of our approach involves learning a multimodal latent dynamic representation. We demonstrate the effectiveness of our method in the context of an electronic parts insertion task, which is challenging for RL agents due to the diverse physical properties of the non-standardized components, as well as simple 3D printed blocks insertion. Furthermore, we evaluate the transferability of MVDeepMDP and analyze the impact of the balancing mechanism of the \u0000<italic>generalized Product-of-Experts</i>\u0000 (gPoE), which is used to combine observable modalities. Finally, we explore the influence of separately processing state modalities of different physical quantities, such as pose and 6D force/torque (F/T) data.","PeriodicalId":13241,"journal":{"name":"IEEE Robotics and Automation Letters","volume":"9 12","pages":"11297-11304"},"PeriodicalIF":4.6,"publicationDate":"2024-10-29","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142645555","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-10-29DOI: 10.1109/LRA.2024.3487491
Pu Feng;Rongye Shi;Size Wang;Junkang Liang;Xin Yu;Simin Li;Wenjun Wu
Reinforcement learning (RL) has shown great promise in addressing multi-agent collision avoidance challenges. However, existing RL-based methods often suffer from low training efficiency and poor action safety. To tackle these issues, we introduce a physics-informed reinforcement learning framework equipped with two modules: a Potential Field (PF) module and a Multi-Agent Multi-Level Safety (MAMLS) module. The PF module uses the Artificial Potential Field method to compute a regularization loss, adaptively integrating it into the critic's loss to enhance training efficiency. The MAMLS module formulates action safety as a constrained optimization problem, deriving safe actions by solving this optimization. Furthermore, to better address the characteristics of multi-agent collision avoidance tasks, multi-agent multi-level constraints are introduced. The results of simulations and real-world experiments showed that our physics-informed framework offers a significant improvement in terms of both the efficiency of training and safety-related metrics over advanced baseline methods.
{"title":"Safe and Efficient Multi-Agent Collision Avoidance With Physics-Informed Reinforcement Learning","authors":"Pu Feng;Rongye Shi;Size Wang;Junkang Liang;Xin Yu;Simin Li;Wenjun Wu","doi":"10.1109/LRA.2024.3487491","DOIUrl":"https://doi.org/10.1109/LRA.2024.3487491","url":null,"abstract":"Reinforcement learning (RL) has shown great promise in addressing multi-agent collision avoidance challenges. However, existing RL-based methods often suffer from low training efficiency and poor action safety. To tackle these issues, we introduce a physics-informed reinforcement learning framework equipped with two modules: a Potential Field (PF) module and a Multi-Agent Multi-Level Safety (MAMLS) module. The PF module uses the Artificial Potential Field method to compute a regularization loss, adaptively integrating it into the critic's loss to enhance training efficiency. The MAMLS module formulates action safety as a constrained optimization problem, deriving safe actions by solving this optimization. Furthermore, to better address the characteristics of multi-agent collision avoidance tasks, multi-agent multi-level constraints are introduced. The results of simulations and real-world experiments showed that our physics-informed framework offers a significant improvement in terms of both the efficiency of training and safety-related metrics over advanced baseline methods.","PeriodicalId":13241,"journal":{"name":"IEEE Robotics and Automation Letters","volume":"9 12","pages":"11138-11145"},"PeriodicalIF":4.6,"publicationDate":"2024-10-29","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142598643","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-10-29DOI: 10.1109/LRA.2024.3487503
Bach-Thuan Bui;Huy-Hoang Bui;Dinh-Tuan Tran;Joo-Ho Lee
State-of-the-art visual localization methods mostly rely on complex procedures to match local descriptors and 3D point clouds. However, these procedures can incur significant costs in terms of inference, storage, and updates over time. In this study, we propose a direct learning-based approach that utilizes a simple network named D2S to represent complex local descriptors and their scene coordinates. Our method is characterized by its simplicity and cost-effectiveness. It solely leverages a single RGB image for localization during the testing phase and only requires a lightweight model to encode a complex sparse scene. The proposed D2S employs a combination of a simple loss function and graph attention to selectively focus on robust descriptors while disregarding areas such as clouds, trees, and several dynamic objects. This selective attention enables D2S to effectively perform a binary-semantic classification for sparse descriptors. Additionally, we propose a simple outdoor dataset to evaluate the capabilities of visual localization methods in scene-specific generalization and self-updating from unlabeled observations. Our approach outperforms the previous regression-based methods in both indoor and outdoor environments. It demonstrates the ability to generalize beyond training data, including scenarios involving transitions from day to night and adapting to domain shifts.
{"title":"D2S: Representing Sparse Descriptors and 3D Coordinates for Camera Relocalization","authors":"Bach-Thuan Bui;Huy-Hoang Bui;Dinh-Tuan Tran;Joo-Ho Lee","doi":"10.1109/LRA.2024.3487503","DOIUrl":"https://doi.org/10.1109/LRA.2024.3487503","url":null,"abstract":"State-of-the-art visual localization methods mostly rely on complex procedures to match local descriptors and 3D point clouds. However, these procedures can incur significant costs in terms of inference, storage, and updates over time. In this study, we propose a direct learning-based approach that utilizes a simple network named D2S to represent complex local descriptors and their scene coordinates. Our method is characterized by its simplicity and cost-effectiveness. It solely leverages a single RGB image for localization during the testing phase and only requires a lightweight model to encode a complex sparse scene. The proposed D2S employs a combination of a simple loss function and graph attention to selectively focus on robust descriptors while disregarding areas such as clouds, trees, and several dynamic objects. This selective attention enables D2S to effectively perform a binary-semantic classification for sparse descriptors. Additionally, we propose a simple outdoor dataset to evaluate the capabilities of visual localization methods in scene-specific generalization and self-updating from unlabeled observations. Our approach outperforms the previous regression-based methods in both indoor and outdoor environments. It demonstrates the ability to generalize beyond training data, including scenarios involving transitions from day to night and adapting to domain shifts.","PeriodicalId":13241,"journal":{"name":"IEEE Robotics and Automation Letters","volume":"9 12","pages":"11449-11456"},"PeriodicalIF":4.6,"publicationDate":"2024-10-29","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142636536","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-10-29DOI: 10.1109/LRA.2024.3487506
Jan Ole von Hartz;Tim Welschehold;Abhinav Valada;Joschka Boedecker
Task Parametrized Gaussian Mixture Models (TP-GMM) are a sample-efficient method for learning object-centric robot manipulation tasks. However, there are several open challenges to applying TP-GMMs in the wild. In this work, we tackle three crucial challenges synergistically. First, end-effector velocities are non-Euclidean and thus hard to model using standard GMMs. We thus propose to factorize the robot's end-effector velocity into its direction and magnitude, and model them using Riemannian GMMs. Second, we leverage the factorized velocities to segment and sequence skills from complex demonstration trajectories. Through the segmentation, we further align skill trajectories and hence leverage time as a powerful inductive bias. Third, we present a method to automatically detect relevant task parameters per skill from visual observations. Our approach enables learning complex manipulation tasks from just five demonstrations while using only RGB-D observations. Extensive experimental evaluations on RLBench demonstrate that our approach achieves state-of-the-art performance with 20-fold improved sample efficiency. Our policies generalize across different environments, object instances, and object positions, while the learned skills are reusable.
{"title":"The Art of Imitation: Learning Long-Horizon Manipulation Tasks From Few Demonstrations","authors":"Jan Ole von Hartz;Tim Welschehold;Abhinav Valada;Joschka Boedecker","doi":"10.1109/LRA.2024.3487506","DOIUrl":"https://doi.org/10.1109/LRA.2024.3487506","url":null,"abstract":"Task Parametrized Gaussian Mixture Models (TP-GMM) are a sample-efficient method for learning object-centric robot manipulation tasks. However, there are several open challenges to applying TP-GMMs in the wild. In this work, we tackle three crucial challenges synergistically. First, end-effector velocities are non-Euclidean and thus hard to model using standard GMMs. We thus propose to factorize the robot's end-effector velocity into its direction and magnitude, and model them using Riemannian GMMs. Second, we leverage the factorized velocities to segment and sequence skills from complex demonstration trajectories. Through the segmentation, we further align skill trajectories and hence leverage time as a powerful inductive bias. Third, we present a method to automatically detect relevant task parameters \u0000<italic>per</i>\u0000 skill from visual observations. Our approach enables learning complex manipulation tasks from just five demonstrations while using only RGB-D observations. Extensive experimental evaluations on RLBench demonstrate that our approach achieves state-of-the-art performance with 20-fold improved sample efficiency. Our policies generalize across different environments, object instances, and object positions, while the learned skills are reusable.","PeriodicalId":13241,"journal":{"name":"IEEE Robotics and Automation Letters","volume":"9 12","pages":"11369-11376"},"PeriodicalIF":4.6,"publicationDate":"2024-10-29","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142636412","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Robot-assisted minimally invasive surgery (RAMIS) provides substantial benefits over traditional open and laparoscopic methods. However, a significant limitation of robot-assisted minimally invasive surgery (RAMIS) is the surgeon's inability to palpate tissues, a crucial technique for examining tissue properties and detecting abnormalities, restricting the widespread adoption of RAMIS. To overcome this obstacle, we introduce MiniTac, a novel vision-based tactile sensor with an ultra-compact cross-sectional diameter of 8mm, designed for seamless integration into mainstream RAMIS devices, particularly the Da Vinci surgical systems. MiniTac features a novel mechanoresponsive photonic elastomer membrane that changes color distribution under varying contact pressures. This color change is captured by an embedded miniature camera, allowing MiniTac to detect tumors both on the tissue surface and in deeper layers typically obscured from endoscopic view. MiniTac's efficacy has been rigorously tested on both phantoms and ex-vivo tissues. By leveraging advanced mechanoresponsive photonic materials, MiniTac represents a significant advancement in integrating tactile sensing into RAMIS, potentially expanding its applicability to a wider array of clinical scenarios that currently rely on traditional surgical approaches.
{"title":"MiniTac: An Ultra-Compact $text{8 mm}$ Vision-Based Tactile Sensor for Enhanced Palpation in Robot-Assisted Minimally Invasive Surgery","authors":"Wanlin Li;Zihang Zhao;Leiyao Cui;Weiyi Zhang;Hangxin Liu;Li-An Li;Yixin Zhu","doi":"10.1109/LRA.2024.3487516","DOIUrl":"https://doi.org/10.1109/LRA.2024.3487516","url":null,"abstract":"Robot-assisted minimally invasive surgery (RAMIS) provides substantial benefits over traditional open and laparoscopic methods. However, a significant limitation of robot-assisted minimally invasive surgery (RAMIS) is the surgeon's inability to palpate tissues, a crucial technique for examining tissue properties and detecting abnormalities, restricting the widespread adoption of RAMIS. To overcome this obstacle, we introduce MiniTac, a novel vision-based tactile sensor with an ultra-compact cross-sectional diameter of 8mm, designed for seamless integration into mainstream RAMIS devices, particularly the Da Vinci surgical systems. MiniTac features a novel mechanoresponsive photonic elastomer membrane that changes color distribution under varying contact pressures. This color change is captured by an embedded miniature camera, allowing MiniTac to detect tumors both on the tissue surface and in deeper layers typically obscured from endoscopic view. MiniTac's efficacy has been rigorously tested on both phantoms and ex-vivo tissues. By leveraging advanced mechanoresponsive photonic materials, MiniTac represents a significant advancement in integrating tactile sensing into RAMIS, potentially expanding its applicability to a wider array of clinical scenarios that currently rely on traditional surgical approaches.","PeriodicalId":13241,"journal":{"name":"IEEE Robotics and Automation Letters","volume":"9 12","pages":"11170-11177"},"PeriodicalIF":4.6,"publicationDate":"2024-10-29","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142598672","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-10-29DOI: 10.1109/LRA.2024.3487512
Kailuan Tang;Shaowu Tang;Chenghua Lu;Shijian Wu;Sicong Liu;Juan Yi;Jian S. Dai;Zheng Wang
Interactions with environmental objects can induce substantial alterations in both exteroceptive and proprioceptive signals. However, the deployment of exteroceptive sensors within underwater soft manipulators encounters numerous challenges and constraints, thereby imposing limitations on their perception capabilities. In this article, we present a novel learning-based exteroceptive approach that utilizes internal proprioceptive signals and harnesses the principles of soft actuator network (SAN). Deformation and vibration resulting from external collisions tend to propagate through the SANs in underwater soft manipulators and can be detected by proprioceptive sensors. We extract features from the sensor signals and develop a fully-connected neural network (FCNN)-based classifier to determine collision positions. We have constructed a training dataset and an independent validation dataset for the purpose of training and validating the classifier. The experimental results affirm that the proposed method can identify collision locations with an accuracy level of 97.11% using the independent validation dataset, which exhibits potential applications within the domain of underwater soft robotics perception and control.
与环境物体的相互作用会引起外部感觉和本体感觉信号的巨大变化。然而，在水下软机械手中部署外感知传感器会遇到许多挑战和限制，从而对其感知能力造成限制。在这篇文章中，我们提出了一种基于学习的新型外感知方法，它利用内部本体感觉信号和软致动器网络（SAN）原理。外部碰撞产生的变形和振动往往会通过水下软体机械手的 SAN 传播，并可被本体感觉传感器检测到。我们从传感器信号中提取特征，并开发了基于全连接神经网络（FCNN）的分类器来确定碰撞位置。我们构建了一个训练数据集和一个独立的验证数据集，用于训练和验证分类器。实验结果表明，利用独立验证数据集，所提出的方法能够以 97.11% 的准确率识别碰撞位置，在水下软机器人感知和控制领域具有潜在的应用前景。
{"title":"Learning Based Exteroception of Soft Underwater Manipulator With Soft Actuator Network","authors":"Kailuan Tang;Shaowu Tang;Chenghua Lu;Shijian Wu;Sicong Liu;Juan Yi;Jian S. Dai;Zheng Wang","doi":"10.1109/LRA.2024.3487512","DOIUrl":"https://doi.org/10.1109/LRA.2024.3487512","url":null,"abstract":"Interactions with environmental objects can induce substantial alterations in both exteroceptive and proprioceptive signals. However, the deployment of exteroceptive sensors within underwater soft manipulators encounters numerous challenges and constraints, thereby imposing limitations on their perception capabilities. In this article, we present a novel learning-based exteroceptive approach that utilizes internal proprioceptive signals and harnesses the principles of soft actuator network (SAN). Deformation and vibration resulting from external collisions tend to propagate through the SANs in underwater soft manipulators and can be detected by proprioceptive sensors. We extract features from the sensor signals and develop a fully-connected neural network (FCNN)-based classifier to determine collision positions. We have constructed a training dataset and an independent validation dataset for the purpose of training and validating the classifier. The experimental results affirm that the proposed method can identify collision locations with an accuracy level of 97.11% using the independent validation dataset, which exhibits potential applications within the domain of underwater soft robotics perception and control.","PeriodicalId":13241,"journal":{"name":"IEEE Robotics and Automation Letters","volume":"9 12","pages":"11082-11089"},"PeriodicalIF":4.6,"publicationDate":"2024-10-29","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142600249","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
This letter presents Open-Structure, a novel benchmark dataset for evaluating visual odometry and SLAM methods. Compared to existing public datasets that primarily offer raw images, Open-Structure provides direct access to point and line measurements, correspondences, structural associations, and co-visibility factor graphs, which can be fed to various stages of SLAM pipelines to mitigate the impact of data preprocessing modules in ablation experiments. The dataset comprises two distinct types of sequences from the perspective of scenarios. The first type maintains reasonable observation and occlusion relationships, as these critical elements are extracted from public image-based sequences using our dataset generator. In contrast, the second type consists of carefully designed simulation sequences that enhance dataset diversity by introducing a wide range of trajectories and observations. Furthermore, a baseline is proposed using our dataset to evaluate widely used modules, including camera pose tracking, parametrization, and factor graph optimization, within SLAM systems. By evaluating these state-of-the-art algorithms across different scenarios, we discern each module's strengths and weaknesses in the context of camera tracking and optimization processes.
本文介绍了用于评估视觉里程测量和 SLAM 方法的新型基准数据集 Open-Structure。与主要提供原始图像的现有公共数据集相比，Open-Structure 数据集可直接获取点和线的测量结果、对应关系、结构关联和共视因子图，这些数据可输入 SLAM 管道的各个阶段，以减轻消融实验中数据预处理模块的影响。从场景的角度来看，该数据集包括两种不同类型的序列。第一种类型保持了合理的观察和遮挡关系，因为这些关键要素是利用我们的数据集生成器从基于图像的公共序列中提取的。相比之下，第二种类型由精心设计的模拟序列组成，通过引入各种轨迹和观测数据来增强数据集的多样性。此外，我们还提出了一个基线，利用我们的数据集来评估 SLAM 系统中广泛使用的模块，包括相机姿态跟踪、参数化和因子图优化。通过在不同场景下对这些先进算法进行评估，我们发现了每个模块在摄像机跟踪和优化过程中的优缺点。
{"title":"Open-Structure: Structural Benchmark Dataset for SLAM Algorithms","authors":"Yanyan Li;Zhao Guo;Ze Yang;Yanbiao Sun;Liang Zhao;Federico Tombari","doi":"10.1109/LRA.2024.3487071","DOIUrl":"https://doi.org/10.1109/LRA.2024.3487071","url":null,"abstract":"This letter presents Open-Structure, a novel benchmark dataset for evaluating visual odometry and SLAM methods. Compared to existing public datasets that primarily offer raw images, Open-Structure provides direct access to point and line measurements, correspondences, structural associations, and co-visibility factor graphs, which can be fed to various stages of SLAM pipelines to mitigate the impact of data preprocessing modules in ablation experiments. The dataset comprises two distinct types of sequences from the perspective of scenarios. The first type maintains reasonable observation and occlusion relationships, as these critical elements are extracted from public image-based sequences using our dataset generator. In contrast, the second type consists of carefully designed simulation sequences that enhance dataset diversity by introducing a wide range of trajectories and observations. Furthermore, a baseline is proposed using our dataset to evaluate widely used modules, including camera pose tracking, parametrization, and factor graph optimization, within SLAM systems. By evaluating these state-of-the-art algorithms across different scenarios, we discern each module's strengths and weaknesses in the context of camera tracking and optimization processes.","PeriodicalId":13241,"journal":{"name":"IEEE Robotics and Automation Letters","volume":"9 12","pages":"11457-11464"},"PeriodicalIF":4.6,"publicationDate":"2024-10-29","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142636538","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Complex operational scenarios increasingly demand that industrial robots sequentially resolve multiple interrelated problems to accomplish complex operational tasks, necessitating robots to have the capacity for not only learning through interaction with the environment but also for continual learning. Current deep reinforcement learning methods have demonstrated substantial prowess in enabling robots to learn individual simple operational skills. However, catastrophic forgetting regarding the continual learning of various distinct tasks under a unified control policy remains a challenge. The lengthy sequential decision-making trajectory in reinforcement learning scenarios results in a massive state-action search space for the agent. Moreover, low-value state-action samples exacerbate the difficulty of continuous learning in reinforcement learning problems. In this letter, we propose a Continual Reinforcement Learning (CRL) method that accommodates the incremental multiskill learning demands of robots. We transform the tightly coupled structure in Guided Policy Search (GPS) algorithms, which closely intertwine local and global policies, into a loosely coupled structure. This revised structure updates the global policy only after the local policy for a specific task has converged, enabling online learning. In incrementally learning new tasks, the global policy is updated using hard parameter sharing and Memory Aware Synapses (MAS), creating task-specific layers while penalizing significant parameter changes in shared layers linked to prior tasks. This method reduces overfitting and mitigates catastrophic forgetting in robotic CRL. We validate our method on PR2, UR5 and Sawyer robots in simulators as well as on a real UR5 robot.
{"title":"Mitigating Catastrophic Forgetting in Robot Continual Learning: A Guided Policy Search Approach Enhanced With Memory-Aware Synapses","authors":"Qingwei Dong;Peng Zeng;Yunpeng He;Guangxi Wan;Xiaoting Dong","doi":"10.1109/LRA.2024.3487484","DOIUrl":"https://doi.org/10.1109/LRA.2024.3487484","url":null,"abstract":"Complex operational scenarios increasingly demand that industrial robots sequentially resolve multiple interrelated problems to accomplish complex operational tasks, necessitating robots to have the capacity for not only learning through interaction with the environment but also for continual learning. Current deep reinforcement learning methods have demonstrated substantial prowess in enabling robots to learn individual simple operational skills. However, catastrophic forgetting regarding the continual learning of various distinct tasks under a unified control policy remains a challenge. The lengthy sequential decision-making trajectory in reinforcement learning scenarios results in a massive state-action search space for the agent. Moreover, low-value state-action samples exacerbate the difficulty of continuous learning in reinforcement learning problems. In this letter, we propose a Continual Reinforcement Learning (CRL) method that accommodates the incremental multiskill learning demands of robots. We transform the tightly coupled structure in Guided Policy Search (GPS) algorithms, which closely intertwine local and global policies, into a loosely coupled structure. This revised structure updates the global policy only after the local policy for a specific task has converged, enabling online learning. In incrementally learning new tasks, the global policy is updated using hard parameter sharing and Memory Aware Synapses (MAS), creating task-specific layers while penalizing significant parameter changes in shared layers linked to prior tasks. This method reduces overfitting and mitigates catastrophic forgetting in robotic CRL. We validate our method on PR2, UR5 and Sawyer robots in simulators as well as on a real UR5 robot.","PeriodicalId":13241,"journal":{"name":"IEEE Robotics and Automation Letters","volume":"9 12","pages":"11242-11249"},"PeriodicalIF":4.6,"publicationDate":"2024-10-29","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142598646","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-10-29DOI: 10.1109/LRA.2024.3487496
Charles Hamesse;Michiel Vlaminck;Hiep Luong;Rob Haelterman
We propose the D epth- V isual- I nertial (DVI) mapping system: a robust multi-sensor fusion framework for dense 3D mapping using time-of-flight cameras equipped with RGB and IMU sensors. Inspired by recent developments in real-time LiDAR-based odometry and mapping, our system uses an error-state iterative Kalman filter for state estimation: it processes the inertial sensor's data for state propagation, followed by a state update first using visual-inertial odometry, then depth-based odometry. This sensor fusion scheme makes our system robust to degenerate scenarios (e.g. lack of visual or geometrical features, fast rotations) and to noisy sensor data, like those that can be obtained with off-the-shelf time-of-flight DVI sensors. For evaluation, we propose the new Bunker DVI Dataset, featuring data from multiple DVI sensors recorded in challenging conditions reflecting search-and-rescue operations. We show the superior robustness and precision of our method against previous work. Following the open science principle, we make both our source code and dataset publicly available.
我们提出了深度-视觉-惯性（DVI）测绘系统：这是一种稳健的多传感器融合框架，用于使用配备了 RGB 和 IMU 传感器的飞行时间照相机进行密集 3D 测绘。受基于激光雷达的实时里程测量和制图的最新发展的启发，我们的系统使用误差状态迭代卡尔曼滤波器进行状态估计：它处理惯性传感器的数据进行状态传播，然后首先使用视觉惯性里程测量进行状态更新，接着使用基于深度的里程测量进行状态更新。这种传感器融合方案使我们的系统对退化场景（如缺乏视觉或几何特征、快速旋转）和嘈杂的传感器数据（如使用现成的飞行时间 DVI 传感器获得的数据）具有鲁棒性。为了进行评估，我们提出了新的掩体 DVI 数据集，该数据集由多个 DVI 传感器在具有挑战性的条件下记录的数据组成，反映了搜救行动的情况。与之前的研究相比，我们的方法具有更高的鲁棒性和精确性。遵循开放科学原则，我们公开了源代码和数据集。
{"title":"Depth-Visual-Inertial (DVI) Mapping System for Robust Indoor 3D Reconstruction","authors":"Charles Hamesse;Michiel Vlaminck;Hiep Luong;Rob Haelterman","doi":"10.1109/LRA.2024.3487496","DOIUrl":"https://doi.org/10.1109/LRA.2024.3487496","url":null,"abstract":"We propose the \u0000<underline>D</u>\u0000epth-\u0000<underline>V</u>\u0000isual-\u0000<underline>I</u>\u0000nertial (DVI) mapping system: a robust multi-sensor fusion framework for dense 3D mapping using time-of-flight cameras equipped with RGB and IMU sensors. Inspired by recent developments in real-time LiDAR-based odometry and mapping, our system uses an error-state iterative Kalman filter for state estimation: it processes the inertial sensor's data for state propagation, followed by a state update first using visual-inertial odometry, then depth-based odometry. This sensor fusion scheme makes our system robust to degenerate scenarios (e.g. lack of visual or geometrical features, fast rotations) and to noisy sensor data, like those that can be obtained with off-the-shelf time-of-flight DVI sensors. For evaluation, we propose the new Bunker DVI Dataset, featuring data from multiple DVI sensors recorded in challenging conditions reflecting search-and-rescue operations. We show the superior robustness and precision of our method against previous work. Following the open science principle, we make both our source code and dataset publicly available.","PeriodicalId":13241,"journal":{"name":"IEEE Robotics and Automation Letters","volume":"9 12","pages":"11313-11320"},"PeriodicalIF":4.6,"publicationDate":"2024-10-29","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142600124","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
京公网安备 11010802042870号
京ICP备2023020795号-1
扫码关注我们
求助内容：
应助结果提醒方式：