Pub Date : 2024-10-21DOI: 10.1109/LRA.2024.3484153
Johan Hatleskog;Kostas Alexis
Degeneracies arising from uninformative geometry are known to deteriorate LiDAR-based localization and mapping. This work introduces a new probabilistic method to detect and mitigate the effect of degeneracies in point-to-plane error minimization. The noise on the Hessian of the point-to-plane optimization problem is characterized by the noise on points and surface normals used in its construction. We exploit this characterization to quantify the probability of a direction being degenerate. The degeneracy-detection procedure is used in a new real-time degeneracy-aware iterative closest point algorithm for LiDAR registration, in which we smoothly attenuate updates in degenerate directions. The method's parameters are selected based on the noise characteristics provided in the LiDAR's datasheet. We validate the approach in four real-world experiments, demonstrating that it outperforms state-of-the-art methods at detecting and mitigating the adverse effects of degeneracies.
{"title":"Probabilistic Degeneracy Detection for Point-to-Plane Error Minimization","authors":"Johan Hatleskog;Kostas Alexis","doi":"10.1109/LRA.2024.3484153","DOIUrl":"https://doi.org/10.1109/LRA.2024.3484153","url":null,"abstract":"Degeneracies arising from uninformative geometry are known to deteriorate LiDAR-based localization and mapping. This work introduces a new probabilistic method to detect and mitigate the effect of degeneracies in point-to-plane error minimization. The noise on the Hessian of the point-to-plane optimization problem is characterized by the noise on points and surface normals used in its construction. We exploit this characterization to quantify the probability of a direction being degenerate. The degeneracy-detection procedure is used in a new real-time degeneracy-aware iterative closest point algorithm for LiDAR registration, in which we smoothly attenuate updates in degenerate directions. The method's parameters are selected based on the noise characteristics provided in the LiDAR's datasheet. We validate the approach in four real-world experiments, demonstrating that it outperforms state-of-the-art methods at detecting and mitigating the adverse effects of degeneracies.","PeriodicalId":13241,"journal":{"name":"IEEE Robotics and Automation Letters","volume":"9 12","pages":"11234-11241"},"PeriodicalIF":4.6,"publicationDate":"2024-10-21","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142598631","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-10-21DOI: 10.1109/LRA.2024.3484131
Yunhai Wang;Lei Yang;Peng Zhou;Jiaming Qi;Liang Lu;Jihong Zhu;Jia Pan
Fabrics present significant challenges to robotic manipulation due to their complex dynamics and infinite degrees of freedom. This letter proposes a non-prehensile approach to aligning a fabric cut piece to a specified target pose, which is a common step for many garment manufacturing tasks. Compared to widely explored prehensile manipulation that grasps and lifts the fabric, the proposed approach uses pushing actions and allows for efficient fabric repositioning with a simpler end-effector. To handle the possible deformation caused by the pushing actions, we introduce the Deformation-aware Rapidly-exploring Random Tree Star (D-RRT*) algorithm that strategically plans the contact poses and actions to slide the fabric to desired configurations by leveraging the friction. Our D-RRT* algorithm enhances fabric manipulation by incorporating deformation-aware action sampling into the path planning process, enabling accurate predictions of non-prehensile actions and efficient navigation through the fabric's configuration space. Extensive simulations and real-world experiments on repositioning fabrics of various shapes and materials demonstrate the effectiveness of the proposed pipeline in achieving stable and efficient manipulation.
{"title":"Efficient Planar Fabric Repositioning: Deformation-Aware RRT* for Non-Prehensile Fabric Manipulation","authors":"Yunhai Wang;Lei Yang;Peng Zhou;Jiaming Qi;Liang Lu;Jihong Zhu;Jia Pan","doi":"10.1109/LRA.2024.3484131","DOIUrl":"https://doi.org/10.1109/LRA.2024.3484131","url":null,"abstract":"Fabrics present significant challenges to robotic manipulation due to their complex dynamics and infinite degrees of freedom. This letter proposes a non-prehensile approach to aligning a fabric cut piece to a specified target pose, which is a common step for many garment manufacturing tasks. Compared to widely explored prehensile manipulation that grasps and lifts the fabric, the proposed approach uses pushing actions and allows for efficient fabric repositioning with a simpler end-effector. To handle the possible deformation caused by the pushing actions, we introduce the Deformation-aware Rapidly-exploring Random Tree Star (D-RRT*) algorithm that strategically plans the contact poses and actions to slide the fabric to desired configurations by leveraging the friction. Our D-RRT* algorithm enhances fabric manipulation by incorporating deformation-aware action sampling into the path planning process, enabling accurate predictions of non-prehensile actions and efficient navigation through the fabric's configuration space. Extensive simulations and real-world experiments on repositioning fabrics of various shapes and materials demonstrate the effectiveness of the proposed pipeline in achieving stable and efficient manipulation.","PeriodicalId":13241,"journal":{"name":"IEEE Robotics and Automation Letters","volume":"9 12","pages":"11258-11265"},"PeriodicalIF":4.6,"publicationDate":"2024-10-21","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=10723787","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142598677","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Adaptive climbing on different surfaces is a great challenge for conventional robots due to a lack of self-sensing capabilities. Inspired by the exceptional sensing ability of feline soles, this study proposes a quadrupedal climbing robot based on self-sensing spiny-claw soles. First, a spiny-claw sole was designed by embedding stainless steel spines into a soft substrate. Next, a tensile strain sensor was designed based on carbon nanotubes and carbonyl iron powder through the squash method and then was integrated into the spiny-claw sole to fabricate the self-sensing sole. Then, a quadrupedal climbing robot was designed using four self-sensing spiny-claw soles. Subsequently, the control strategy of the self-sensing climbing robot was designed. Finally, the climbing performance of the self-sensing robot was experimentally tested. It is demonstrated that the robot can climb on different inclined surfaces with an angle of 0 $^circ$ to 75 $^circ$ and on three different rough surfaces. In addition, the maximum load of the robot is 175 g when climbing on a 45 $^circ$ inclined surface. More importantly, the robot can detect whether there is an obstacle in the climbing path through the self-sensing soles and perform appropriate obstacle avoidance operations accordingly.
{"title":"A Spiny Claws Climbing Robot Based on Self-Sensing Soles","authors":"Rui Li;Changze Wu;Shuang Yan;Chuan Li;Xinglong Gong;Chul-Hee Lee;Mengjie Shou","doi":"10.1109/LRA.2024.3484179","DOIUrl":"https://doi.org/10.1109/LRA.2024.3484179","url":null,"abstract":"Adaptive climbing on different surfaces is a great challenge for conventional robots due to a lack of self-sensing capabilities. Inspired by the exceptional sensing ability of feline soles, this study proposes a quadrupedal climbing robot based on self-sensing spiny-claw soles. First, a spiny-claw sole was designed by embedding stainless steel spines into a soft substrate. Next, a tensile strain sensor was designed based on carbon nanotubes and carbonyl iron powder through the squash method and then was integrated into the spiny-claw sole to fabricate the self-sensing sole. Then, a quadrupedal climbing robot was designed using four self-sensing spiny-claw soles. Subsequently, the control strategy of the self-sensing climbing robot was designed. Finally, the climbing performance of the self-sensing robot was experimentally tested. It is demonstrated that the robot can climb on different inclined surfaces with an angle of 0\u0000<inline-formula><tex-math>$^circ$</tex-math></inline-formula>\u0000 to 75\u0000<inline-formula><tex-math>$^circ$</tex-math></inline-formula>\u0000 and on three different rough surfaces. In addition, the maximum load of the robot is 175 g when climbing on a 45\u0000<inline-formula><tex-math>$^circ$</tex-math></inline-formula>\u0000 inclined surface. More importantly, the robot can detect whether there is an obstacle in the climbing path through the self-sensing soles and perform appropriate obstacle avoidance operations accordingly.","PeriodicalId":13241,"journal":{"name":"IEEE Robotics and Automation Letters","volume":"9 12","pages":"10906-10913"},"PeriodicalIF":4.6,"publicationDate":"2024-10-21","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142524208","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-10-21DOI: 10.1109/LRA.2024.3484157
Qingkai Guo;Yu Sun;Zipan Zhao;Jiajia Ning;Ling Wang;Yuxin Lv;Xuefeng Chen;Laihao Yang
In the realm of robotics, suction-based adhesion plays a pivotal role in applications ranging from object transfer to wall-climbing robots. To improve the sealing and attachment stability of suction cups, researchers have employed state-of-the-art techniques, including the use of soft materials with better conformal properties or jamming mechanisms. However, soft materials can cause undesired deformation of the suction cup under load, and the use of jamming mechanisms has limitations in terms of size and weight. This letter introduces a novel Rigid-Soft Hybrid Suction Cup (RSH-SC) designed for enhanced stability and energy-efficient attachment in nonideal conditions (i.e., suction on irregular and inclined surfaces). To emulate the octopus's dexterous suction and the limpet's robust adhesion capabilities, the RSH-SC integrates a rigid shell for better sealing and torque resistance. Notably, compared to a soft suction cup made from Ecoflex 00-50, the RSH-SC's torque deformation resistance is 550 times greater. The RSH-SC's unique structure also allows it to maintain secure attachment without continuous vacuum pressure, thus conserving energy. A crawling robot utilizing RSH-SCs showcased stable movement on ceiling, which can significantly advance the capabilities of soft robots in complex environments, paving the way for broader applications in robotics and automated systems.
{"title":"Rigid-Soft Hybrid Suction Cups for Enhanced Anti-Torque and Energy-Efficient Attachment","authors":"Qingkai Guo;Yu Sun;Zipan Zhao;Jiajia Ning;Ling Wang;Yuxin Lv;Xuefeng Chen;Laihao Yang","doi":"10.1109/LRA.2024.3484157","DOIUrl":"https://doi.org/10.1109/LRA.2024.3484157","url":null,"abstract":"In the realm of robotics, suction-based adhesion plays a pivotal role in applications ranging from object transfer to wall-climbing robots. To improve the sealing and attachment stability of suction cups, researchers have employed state-of-the-art techniques, including the use of soft materials with better conformal properties or jamming mechanisms. However, soft materials can cause undesired deformation of the suction cup under load, and the use of jamming mechanisms has limitations in terms of size and weight. This letter introduces a novel Rigid-Soft Hybrid Suction Cup (RSH-SC) designed for enhanced stability and energy-efficient attachment in nonideal conditions (i.e., suction on irregular and inclined surfaces). To emulate the octopus's dexterous suction and the limpet's robust adhesion capabilities, the RSH-SC integrates a rigid shell for better sealing and torque resistance. Notably, compared to a soft suction cup made from Ecoflex 00-50, the RSH-SC's torque deformation resistance is 550 times greater. The RSH-SC's unique structure also allows it to maintain secure attachment without continuous vacuum pressure, thus conserving energy. A crawling robot utilizing RSH-SCs showcased stable movement on ceiling, which can significantly advance the capabilities of soft robots in complex environments, paving the way for broader applications in robotics and automated systems.","PeriodicalId":13241,"journal":{"name":"IEEE Robotics and Automation Letters","volume":"9 12","pages":"10970-10977"},"PeriodicalIF":4.6,"publicationDate":"2024-10-21","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142565501","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-10-21DOI: 10.1109/LRA.2024.3484167
Ji-Heon Oh;Ismael Espinoza;Danbi Jung;Tae-Seong Kim
Dual-arm robots can perform bimanual long-horizon (LH) manipulation, surpassing the capabilities of single-arm robots. However, bimanual LH tasks are challenging for robot intelligence due to the complexity of long sequence variables and multi-agent interactions. While Multi-Agent Reinforcement Learning (MARL) has shown promising results in agent interactions, these models struggle with sequential LH tasks due to limitations in credit assignment, vanishing memory, and the exploration-exploitation trade-off. This paper introduces a novel dual-arm robot intelligence framework, Temporal-Context Transformer Reinforcement Learning (TC-TRL), which integrates both a hybrid offline-online policy and imitation learning. TC-TRL leverages the attention mechanism to identify relevant temporal-context information from the LH observations space, updating the encoder value function and generating an optimal actions sequence using a decoder module, which uses demonstration guidance during online training. TC-TRL is tested on six bimanual tasks, and its performance is compared against five baseline RLs: MAPPO, HAPPO, IPPO, MAT, and DA-MAT. The results show that TC-TRL outperforms the three PPO-based RLs with an average success rate of 63.46%, 42.23% against MAT, and 30.91% for DA-MAT.
{"title":"Bimanual Long-Horizon Manipulation Via Temporal-Context Transformer RL","authors":"Ji-Heon Oh;Ismael Espinoza;Danbi Jung;Tae-Seong Kim","doi":"10.1109/LRA.2024.3484167","DOIUrl":"https://doi.org/10.1109/LRA.2024.3484167","url":null,"abstract":"Dual-arm robots can perform bimanual long-horizon (LH) manipulation, surpassing the capabilities of single-arm robots. However, bimanual LH tasks are challenging for robot intelligence due to the complexity of long sequence variables and multi-agent interactions. While Multi-Agent Reinforcement Learning (MARL) has shown promising results in agent interactions, these models struggle with sequential LH tasks due to limitations in credit assignment, vanishing memory, and the exploration-exploitation trade-off. This paper introduces a novel dual-arm robot intelligence framework, Temporal-Context Transformer Reinforcement Learning (TC-TRL), which integrates both a hybrid offline-online policy and imitation learning. TC-TRL leverages the attention mechanism to identify relevant temporal-context information from the LH observations space, updating the encoder value function and generating an optimal actions sequence using a decoder module, which uses demonstration guidance during online training. TC-TRL is tested on six bimanual tasks, and its performance is compared against five baseline RLs: MAPPO, HAPPO, IPPO, MAT, and DA-MAT. The results show that TC-TRL outperforms the three PPO-based RLs with an average success rate of 63.46%, 42.23% against MAT, and 30.91% for DA-MAT.","PeriodicalId":13241,"journal":{"name":"IEEE Robotics and Automation Letters","volume":"9 12","pages":"10898-10905"},"PeriodicalIF":4.6,"publicationDate":"2024-10-21","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142524209","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-10-17DOI: 10.1109/LRA.2024.3483628
Emily Kamienski;Harry Asada
It is often challenging to pick suitable data features for learning problems. Sometimes certain regions of the data are harder to learn because they are not well characterized by the selected data features. The challenge is amplified when resources for sensing and computation are limited and time-critical, yet reliable decisions must be made. For example, a robotic system for preventing falls of elderly people needs a real-time fall predictor, with low false positive and false negative rates, using a simple wearable sensor to activate a fall prevention mechanism. Here we present a methodology for assessing the learnability of data based on the Lipschitz quotient. We develop a procedure for determining which regions of the dataset contain adversarial data points, input data that look similar but belong to different target classes. Regardless of the learning model, it will be hard to learn such data. We then present a method for determining which additional feature(s) are most effective in improving the predictability of each of these regions. This is a model-independent data analysis that can be executed before constructing a prediction model through machine learning or other techniques. We demonstrate this method on two synthetic datasets and a dataset of human falls, which uses inertial measurement unit signals. For the fall dataset, we identified two groups of adversarial data points and improved the predictability of each group over the baseline dataset, as assessed by Lipschitz, by using 2 different sets of features. This work offers a valuable tool for assessing data learnability that can be applied to not only fall prediction problems, but also other robotics applications that learn from data.
{"title":"Model Free Method of Screening Training Data for Adversarial Datapoints Through Local Lipschitz Quotient Analysis","authors":"Emily Kamienski;Harry Asada","doi":"10.1109/LRA.2024.3483628","DOIUrl":"https://doi.org/10.1109/LRA.2024.3483628","url":null,"abstract":"It is often challenging to pick suitable data features for learning problems. Sometimes certain regions of the data are harder to learn because they are not well characterized by the selected data features. The challenge is amplified when resources for sensing and computation are limited and time-critical, yet reliable decisions must be made. For example, a robotic system for preventing falls of elderly people needs a real-time fall predictor, with low false positive and false negative rates, using a simple wearable sensor to activate a fall prevention mechanism. Here we present a methodology for assessing the learnability of data based on the Lipschitz quotient. We develop a procedure for determining which regions of the dataset contain adversarial data points, input data that look similar but belong to different target classes. Regardless of the learning model, it will be hard to learn such data. We then present a method for determining which additional feature(s) are most effective in improving the predictability of each of these regions. This is a model-independent data analysis that can be executed before constructing a prediction model through machine learning or other techniques. We demonstrate this method on two synthetic datasets and a dataset of human falls, which uses inertial measurement unit signals. For the fall dataset, we identified two groups of adversarial data points and improved the predictability of each group over the baseline dataset, as assessed by Lipschitz, by using 2 different sets of features. This work offers a valuable tool for assessing data learnability that can be applied to not only fall prediction problems, but also other robotics applications that learn from data.","PeriodicalId":13241,"journal":{"name":"IEEE Robotics and Automation Letters","volume":"9 12","pages":"11122-11129"},"PeriodicalIF":4.6,"publicationDate":"2024-10-17","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142598638","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-10-17DOI: 10.1109/LRA.2024.3483048
Yuhua Song;Lifeng Zhu;Jinfeng Li;Jiawei Deng;Cheng Wang;Aiguo Song
This letter focuses on the flexible tube of a bronchoscope robot used in pulmonary intervention surgery, which is considered as a continuum robot. The dynamics model is proposed based on the Koopman operator, leveraging real data to solve for the system matrix parameters accurately. To enhance control precision, we designed a model predictive control (MPC) algorithm aimed at tracking the desired curvature and deflection angles of the flexible tube. The MPC controller uses real-time data from electromagnetic sensors to adjust the tube shape, ensuring accurate and responsive manipulation. The effectiveness of the proposed algorithm are validated through extensive experiments conducted on the Binary experimental platform. Compared with the traditional open-loop control method, the proposed algorithm has achieved significant improvements in tracking performance and operational reliability. Each system state error of the proposed algorithm is smaller, which means that better tracking performance can be obtained under the proposed MPC scheme.
{"title":"MPC Design of a Continuum Robot for Pulmonary Interventional Surgery Using Koopman Operators","authors":"Yuhua Song;Lifeng Zhu;Jinfeng Li;Jiawei Deng;Cheng Wang;Aiguo Song","doi":"10.1109/LRA.2024.3483048","DOIUrl":"https://doi.org/10.1109/LRA.2024.3483048","url":null,"abstract":"This letter focuses on the flexible tube of a bronchoscope robot used in pulmonary intervention surgery, which is considered as a continuum robot. The dynamics model is proposed based on the Koopman operator, leveraging real data to solve for the system matrix parameters accurately. To enhance control precision, we designed a model predictive control (MPC) algorithm aimed at tracking the desired curvature and deflection angles of the flexible tube. The MPC controller uses real-time data from electromagnetic sensors to adjust the tube shape, ensuring accurate and responsive manipulation. The effectiveness of the proposed algorithm are validated through extensive experiments conducted on the Binary experimental platform. Compared with the traditional open-loop control method, the proposed algorithm has achieved significant improvements in tracking performance and operational reliability. Each system state error of the proposed algorithm is smaller, which means that better tracking performance can be obtained under the proposed MPC scheme.","PeriodicalId":13241,"journal":{"name":"IEEE Robotics and Automation Letters","volume":"9 12","pages":"10819-10826"},"PeriodicalIF":4.6,"publicationDate":"2024-10-17","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142518206","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Vision-and-Language Navigation (VLN) has garnered widespread attention and research interest due to its potential applications in real-world scenarios. Despite significant progress in the VLN field in recent years, limitations persist. Many agents struggle to make accurate decisions when faced with similar candidate views during navigation, relying solely on the overall features of these views. This challenge primarily arises from the lack of common-sense knowledge about room layouts. Recognizing that room knowledge can establish relationships between rooms and objects in the environment, we construct room layout knowledge described in natural language by leveraging BLIP-2, including relationships between rooms and individual objects, relationships between objects, attributes of individual objects (such as color), and room types, thus providing comprehensive room layout information to the agent. We propose a Knowledge-Enhanced Scene Understanding (KESU) model to augment the agent's understanding of the environment by leveraging room layout knowledge. The Instruction Augmentation Module (IA) and the Knowledge History Fusion Module (KHF) in KESU respectively provide room layout knowledge for instructions and vision-history features, thereby enhancing the agent's navigation abilities. To more effectively integrate knowledge information with instruction features, we introduce Dynamic Residual Fusion (DRF) in the IA module. Finally, we conduct extensive experiments on the R2R, REVERIE, and SOON datasets, demonstrating the effectiveness of the proposed approach.
视觉语言导航（VLN）因其在现实世界中的潜在应用而受到广泛关注和研究兴趣。尽管近年来视觉语言导航领域取得了重大进展，但其局限性依然存在。在导航过程中，面对类似的候选视图，许多代理都很难做出准确的决策，只能依赖于这些视图的整体特征。这一挑战主要源于缺乏有关房间布局的常识性知识。认识到房间知识可以建立房间与环境中物体之间的关系，我们利用 BLIP-2 构建了用自然语言描述的房间布局知识，包括房间与单个物体之间的关系、物体之间的关系、单个物体的属性（如颜色）以及房间类型，从而为代理提供全面的房间布局信息。我们提出了一个知识增强场景理解（KESU）模型，通过利用房间布局知识来增强代理对环境的理解。KESU 中的指令增强模块（IA）和知识历史融合模块（KHF）分别为指令和视觉历史特征提供房间布局知识，从而增强代理的导航能力。为了更有效地整合知识信息与指令特征，我们在 IA 模块中引入了动态残留融合（DRF）。最后，我们在 R2R、REVERIE 和 SOON 数据集上进行了大量实验，证明了所提方法的有效性。
{"title":"Enhancing Scene Understanding for Vision-and-Language Navigation by Knowledge Awareness","authors":"Fang Gao;Jingfeng Tang;Jiabao Wang;Shaodong Li;Jun Yu","doi":"10.1109/LRA.2024.3483042","DOIUrl":"https://doi.org/10.1109/LRA.2024.3483042","url":null,"abstract":"Vision-and-Language Navigation (VLN) has garnered widespread attention and research interest due to its potential applications in real-world scenarios. Despite significant progress in the VLN field in recent years, limitations persist. Many agents struggle to make accurate decisions when faced with similar candidate views during navigation, relying solely on the overall features of these views. This challenge primarily arises from the lack of common-sense knowledge about room layouts. Recognizing that room knowledge can establish relationships between rooms and objects in the environment, we construct room layout knowledge described in natural language by leveraging BLIP-2, including relationships between rooms and individual objects, relationships between objects, attributes of individual objects (such as color), and room types, thus providing comprehensive room layout information to the agent. We propose a Knowledge-Enhanced Scene Understanding (KESU) model to augment the agent's understanding of the environment by leveraging room layout knowledge. The Instruction Augmentation Module (IA) and the Knowledge History Fusion Module (KHF) in KESU respectively provide room layout knowledge for instructions and vision-history features, thereby enhancing the agent's navigation abilities. To more effectively integrate knowledge information with instruction features, we introduce Dynamic Residual Fusion (DRF) in the IA module. Finally, we conduct extensive experiments on the R2R, REVERIE, and SOON datasets, demonstrating the effectiveness of the proposed approach.","PeriodicalId":13241,"journal":{"name":"IEEE Robotics and Automation Letters","volume":"9 12","pages":"10874-10881"},"PeriodicalIF":4.6,"publicationDate":"2024-10-17","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142518184","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
In this work, we introduce a novel multimodal descriptor, the image-assisting binary and triangle combined (iBTC) descriptor, which fuses LiDAR (Light Detection and Ranging) and camera measurements for 3D place recognition. The inherent invariance of a triangle to rigid transformations inspires us to design triangle-based descriptors. We first extract distinct 3D key points from both LiDAR and camera measurements and organize them into triplets to form triangles. By utilizing the lengths of the sides of these triangles, we can create triangle descriptors, enabling the rapid retrieval of similar triangles from a database. By encoding the geometric and visual details at the triangle vertices into binary descriptors, we augment the triangle descriptors with richer local information. This enrichment process empowers our descriptors to reject mis-matched triangle pairs. Consequently, the remaining matched triangle pairs yield accurate loop closure place indices and relative poses. In our experiments, we conduct a thorough comparison of our proposed method with several SOTA methods across public and self-collected datasets. The results demonstrate that our method exhibits superior performance in place recognition and overcomes the limitations associated with the unimodal methods like BTC, RING++, ORB-DBoW2, and NetVLAD. Additionally, we perform a time cost benchmark experiment and the result indicates that our method's time consumption is reasonable, compared with baseline methods.
在这项工作中，我们引入了一种新颖的多模态描述符--图像辅助二进制和三角形组合（iBTC）描述符，它将激光雷达（LiDAR）和照相机测量融合在一起，用于三维地点识别。三角形对刚性变换的固有不变性启发我们设计基于三角形的描述符。我们首先从激光雷达和摄像头的测量结果中提取不同的三维关键点，并将它们组织成三联体，形成三角形。通过利用这些三角形的边长，我们可以创建三角形描述符，从而能够从数据库中快速检索类似的三角形。通过将三角形顶点的几何和视觉细节编码成二进制描述符，我们用更丰富的本地信息增强了三角形描述符。这一丰富过程使我们的描述符能够剔除错误匹配的三角形对。因此，剩余的匹配三角形对可以产生精确的闭合环位置指数和相对位置。在实验中，我们通过公共数据集和自收集数据集将我们提出的方法与几种 SOTA 方法进行了全面比较。结果表明，我们的方法在地点识别方面表现出卓越的性能，克服了 BTC、RING++、ORB-DBoW2 和 NetVLAD 等单模态方法的局限性。此外，我们还进行了时间成本基准实验，结果表明，与基线方法相比，我们的方法耗时合理。
{"title":"iBTC: An Image-Assisting Binary and Triangle Combined Descriptor for Place Recognition by Fusing LiDAR and Camera Measurements","authors":"Zuhao Zou;Chunran Zheng;Chongjian Yuan;Shunbo Zhou;Kaiwen Xue;Fu Zhang","doi":"10.1109/LRA.2024.3483032","DOIUrl":"https://doi.org/10.1109/LRA.2024.3483032","url":null,"abstract":"In this work, we introduce a novel multimodal descriptor, the image-assisting binary and triangle combined (iBTC) descriptor, which fuses LiDAR (Light Detection and Ranging) and camera measurements for 3D place recognition. The inherent invariance of a triangle to rigid transformations inspires us to design triangle-based descriptors. We first extract distinct 3D key points from both LiDAR and camera measurements and organize them into triplets to form triangles. By utilizing the lengths of the sides of these triangles, we can create triangle descriptors, enabling the rapid retrieval of similar triangles from a database. By encoding the geometric and visual details at the triangle vertices into binary descriptors, we augment the triangle descriptors with richer local information. This enrichment process empowers our descriptors to reject mis-matched triangle pairs. Consequently, the remaining matched triangle pairs yield accurate loop closure place indices and relative poses. In our experiments, we conduct a thorough comparison of our proposed method with several SOTA methods across public and self-collected datasets. The results demonstrate that our method exhibits superior performance in place recognition and overcomes the limitations associated with the unimodal methods like BTC, RING++, ORB-DBoW2, and NetVLAD. Additionally, we perform a time cost benchmark experiment and the result indicates that our method's time consumption is reasonable, compared with baseline methods.","PeriodicalId":13241,"journal":{"name":"IEEE Robotics and Automation Letters","volume":"9 12","pages":"10858-10865"},"PeriodicalIF":4.6,"publicationDate":"2024-10-17","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142518168","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-10-17DOI: 10.1109/LRA.2024.3483044
Takamasa Kominami;Kazuhiro Shimonomura
This research proposes a passively driven perching mechanism designed to land not only on flat surface but also on various shaped objects, such as bar, plate, and spherical object, without the use of additional actuators. The perching mechanism consists of three under actuated fingers, and utilizes two passive drive mechanisms: one for landing without bending the fingers but by pinching, and another for grasping by bending the fingers to land. As a result, a lightweight apparatus capable of landing on various objects without the need for additional CPUs, motors, or batteries for driving and controlling was achieved. The gripping force generated by this mechanism was calculated, and it was found that the gripping force varied depending on the size of the object. Finally, the actual forces exerted by the fingers during landing on objects like safety cones, boards, or pipes were measured and compared with theoretical values.
{"title":"Versatile Perching Using a Passive Mechanism With Under Actuated Fingers for Multirotor AAV","authors":"Takamasa Kominami;Kazuhiro Shimonomura","doi":"10.1109/LRA.2024.3483044","DOIUrl":"https://doi.org/10.1109/LRA.2024.3483044","url":null,"abstract":"This research proposes a passively driven perching mechanism designed to land not only on flat surface but also on various shaped objects, such as bar, plate, and spherical object, without the use of additional actuators. The perching mechanism consists of three under actuated fingers, and utilizes two passive drive mechanisms: one for landing without bending the fingers but by pinching, and another for grasping by bending the fingers to land. As a result, a lightweight apparatus capable of landing on various objects without the need for additional CPUs, motors, or batteries for driving and controlling was achieved. The gripping force generated by this mechanism was calculated, and it was found that the gripping force varied depending on the size of the object. Finally, the actual forces exerted by the fingers during landing on objects like safety cones, boards, or pipes were measured and compared with theoretical values.","PeriodicalId":13241,"journal":{"name":"IEEE Robotics and Automation Letters","volume":"9 12","pages":"11226-11233"},"PeriodicalIF":4.6,"publicationDate":"2024-10-17","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142598628","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
京公网安备 11010802042870号
京ICP备2023020795号-1
扫码关注我们
求助内容：
应助结果提醒方式：