Pub Date : 2023-12-04DOI: 10.1109/ROBIO58561.2023.10354989
Jingcheng Jiang, Yifang Zhang, N. Tsagarakis
The timing belt transmission offers numerous advantages for legged robots, including high efficiency, impact absorption and large range of joint motion. However, the transmission error under high load remains challenging to locomotion control and further applications of belt transmission. Traditional linear models cannot effectively model the belt deformation under a wide range of tension variations due to the nonlinearity. In this paper, we propose a model of the compensation for the belt transmission error based on the pretension and torque of the pully. The adopted approach bypasses the complexity of elaborate physical model derivations, yielding a non-linear model for transmission system errors through straightforward fitting. Based on the proposed model, an error compensation control is investigated and tested with an one-DoF leg prototype of legged robot. The alignment between experimental results and theoretical analysis demonstrates the accuracy of the modeling and the effectiveness of the error compensation control method. The proposed model provides a convenient and straightforward solution to effectively compensate for the belt transmission errors in legged robots.
{"title":"Modelling and Compensation for Transmission Error of Timing Belt in Legged Robots","authors":"Jingcheng Jiang, Yifang Zhang, N. Tsagarakis","doi":"10.1109/ROBIO58561.2023.10354989","DOIUrl":"https://doi.org/10.1109/ROBIO58561.2023.10354989","url":null,"abstract":"The timing belt transmission offers numerous advantages for legged robots, including high efficiency, impact absorption and large range of joint motion. However, the transmission error under high load remains challenging to locomotion control and further applications of belt transmission. Traditional linear models cannot effectively model the belt deformation under a wide range of tension variations due to the nonlinearity. In this paper, we propose a model of the compensation for the belt transmission error based on the pretension and torque of the pully. The adopted approach bypasses the complexity of elaborate physical model derivations, yielding a non-linear model for transmission system errors through straightforward fitting. Based on the proposed model, an error compensation control is investigated and tested with an one-DoF leg prototype of legged robot. The alignment between experimental results and theoretical analysis demonstrates the accuracy of the modeling and the effectiveness of the error compensation control method. The proposed model provides a convenient and straightforward solution to effectively compensate for the belt transmission errors in legged robots.","PeriodicalId":505134,"journal":{"name":"2023 IEEE International Conference on Robotics and Biomimetics (ROBIO)","volume":"60 9","pages":"1-6"},"PeriodicalIF":0.0,"publicationDate":"2023-12-04","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"139187064","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2023-12-04DOI: 10.1109/ROBIO58561.2023.10355009
Umer Huzaifa, Dimuthu D. K. Arachchige, Muhammad Aneeq uz Zaman, Usman Syed
Soft robots have shown their value as alternatives or supplements to rigid robots in applications like search and rescue missions and complex precise tasks. Their ability to take on various shapes and apply adaptable force gives them an advantage over stiff robots. However, sometimes their soft structure doesn’t offer enough force for the task. Hybrid soft robots (HSRs) combine a soft body with a stronger backbone to handle tasks needing more strength. This rigid part lets us use rigid body dynamics to estimate HSR behavior. Here, we introduce a simplified N-link rigid body dynamic model with constant stiffness to mimic HSR behavior. While soft robots’ stiffness varies, the backbone in HSRs makes it similar to having constant stiffness. Comparing experiments supports the effectiveness of our N-link model for HSR modeling.
在搜救任务和复杂精密任务等应用中，软体机器人已显示出其作为刚性机器人替代品或补充的价值。与刚性机器人相比，软体机器人能够呈现出各种形状，并能施加适应性强的力，这使它们更具优势。然而，有时它们的软结构并不能为任务提供足够的力。混合软体机器人（HSR）结合了软体和更坚固的骨架，以处理需要更多力量的任务。我们可以利用刚体动力学来估计混合软体机器人的行为。在这里，我们引入了一个简化的具有恒定刚度的 N 连杆刚体动力学模型来模拟 HSR 的行为。软体机器人的刚度是变化的，而 HSR 中的骨架使其类似于具有恒定刚度。对比实验证明了我们的 N-连杆模型在 HSR 建模中的有效性。
{"title":"Simplified Modeling of Hybrid Soft Robots with Constant Stiffness Assumption","authors":"Umer Huzaifa, Dimuthu D. K. Arachchige, Muhammad Aneeq uz Zaman, Usman Syed","doi":"10.1109/ROBIO58561.2023.10355009","DOIUrl":"https://doi.org/10.1109/ROBIO58561.2023.10355009","url":null,"abstract":"Soft robots have shown their value as alternatives or supplements to rigid robots in applications like search and rescue missions and complex precise tasks. Their ability to take on various shapes and apply adaptable force gives them an advantage over stiff robots. However, sometimes their soft structure doesn’t offer enough force for the task. Hybrid soft robots (HSRs) combine a soft body with a stronger backbone to handle tasks needing more strength. This rigid part lets us use rigid body dynamics to estimate HSR behavior. Here, we introduce a simplified N-link rigid body dynamic model with constant stiffness to mimic HSR behavior. While soft robots’ stiffness varies, the backbone in HSRs makes it similar to having constant stiffness. Comparing experiments supports the effectiveness of our N-link model for HSR modeling.","PeriodicalId":505134,"journal":{"name":"2023 IEEE International Conference on Robotics and Biomimetics (ROBIO)","volume":"57 11","pages":"1-6"},"PeriodicalIF":0.0,"publicationDate":"2023-12-04","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"139187133","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Feature point extraction and matching is the key technology in object detection and simultaneous localization and mapping (SLAM). Aiming at the problems such as easy redundancy of feature points extracted by traditional ORB algorithm, low matching accuracy of mainstream robust estimation algorithms and low real-time performance, an improved ORB-GMS image feature extraction and matching algorithm is proposed. Firstly, the algorithm uses the gray value of the image to calculate the adaptive extraction threshold of the feature points. Then the image pyramid is constructed according to the image size. The set number of total feature points to be extracted is evenly distributed to each layer image according to the area ratio; Extract feature points from each layer of the image pyramid, and count the extracted feature points from each layer. If the number of feature points extracted from each layer meets the set number of images from each layer, the extraction ends. Then the quadtree algorithm is used to homogenize the feature points. Finally, the network scoring model is optimized from 8 neighborhood to 4 neighborhood, which reduces the computing time. Experimental results show that the matching accuracy of the proposed algorithm is 14% higher than that of the original algorithm, and the running time is 12% lower.
{"title":"An improved ORB-GMS image feature extraction and matching algorithm*","authors":"Zhiying Tan, Wenbo Fan, Weifeng Kong, Xu Tao, Linsen Xu, Xiaobin Xu","doi":"10.1109/ROBIO58561.2023.10355043","DOIUrl":"https://doi.org/10.1109/ROBIO58561.2023.10355043","url":null,"abstract":"Feature point extraction and matching is the key technology in object detection and simultaneous localization and mapping (SLAM). Aiming at the problems such as easy redundancy of feature points extracted by traditional ORB algorithm, low matching accuracy of mainstream robust estimation algorithms and low real-time performance, an improved ORB-GMS image feature extraction and matching algorithm is proposed. Firstly, the algorithm uses the gray value of the image to calculate the adaptive extraction threshold of the feature points. Then the image pyramid is constructed according to the image size. The set number of total feature points to be extracted is evenly distributed to each layer image according to the area ratio; Extract feature points from each layer of the image pyramid, and count the extracted feature points from each layer. If the number of feature points extracted from each layer meets the set number of images from each layer, the extraction ends. Then the quadtree algorithm is used to homogenize the feature points. Finally, the network scoring model is optimized from 8 neighborhood to 4 neighborhood, which reduces the computing time. Experimental results show that the matching accuracy of the proposed algorithm is 14% higher than that of the original algorithm, and the running time is 12% lower.","PeriodicalId":505134,"journal":{"name":"2023 IEEE International Conference on Robotics and Biomimetics (ROBIO)","volume":"78 4","pages":"1-6"},"PeriodicalIF":0.0,"publicationDate":"2023-12-04","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"139187170","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2023-12-04DOI: 10.1109/ROBIO58561.2023.10354783
Guanhao Xie, Duo Zhao, Qichao Tang, Muhua Zhang, Wenjie Zhao, Yewen Wang
Due to the widespread use of robotic arms, path planning for them has always been a hot research topic. However, traditional path planning algorithms struggle to ensure low disparity in each path, making them unsuitable for operation scenarios with high safety requirements, such as the undercarriage environment of train. A Reinforcement Learning (RL) framework is proposed in this article to address this challenge. The Proximal Policy Optimization (PPO) algorithm has been enhanced, resulting in a variant referred to as Randomized PPO (RPPO), which demonstrates slightly accelerated convergence speed. Additionally, a reward model is proposed to assist the agent in escaping local optima. For modeling application environment, lidar is employed for obtaining obstacle point cloud information, which is then transformed into an octree grid map for maneuvering the robotic arm to avoid obstacles. According to the experimental results, the paths planned by our system are superior to those of RRT* in terms of both average length and standard deviation, and RPPO exhibits better convergence speed and path standard deviation compared to PPO.
{"title":"Path Planning for Robotic Arm Based on Reinforcement Learning under the Train","authors":"Guanhao Xie, Duo Zhao, Qichao Tang, Muhua Zhang, Wenjie Zhao, Yewen Wang","doi":"10.1109/ROBIO58561.2023.10354783","DOIUrl":"https://doi.org/10.1109/ROBIO58561.2023.10354783","url":null,"abstract":"Due to the widespread use of robotic arms, path planning for them has always been a hot research topic. However, traditional path planning algorithms struggle to ensure low disparity in each path, making them unsuitable for operation scenarios with high safety requirements, such as the undercarriage environment of train. A Reinforcement Learning (RL) framework is proposed in this article to address this challenge. The Proximal Policy Optimization (PPO) algorithm has been enhanced, resulting in a variant referred to as Randomized PPO (RPPO), which demonstrates slightly accelerated convergence speed. Additionally, a reward model is proposed to assist the agent in escaping local optima. For modeling application environment, lidar is employed for obtaining obstacle point cloud information, which is then transformed into an octree grid map for maneuvering the robotic arm to avoid obstacles. According to the experimental results, the paths planned by our system are superior to those of RRT* in terms of both average length and standard deviation, and RPPO exhibits better convergence speed and path standard deviation compared to PPO.","PeriodicalId":505134,"journal":{"name":"2023 IEEE International Conference on Robotics and Biomimetics (ROBIO)","volume":"34 12","pages":"1-8"},"PeriodicalIF":0.0,"publicationDate":"2023-12-04","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"139187354","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
This paper presents a general estimation method of deformation for the self-balancing lower limb exoskeleton (SBLLE). In particular, we propose a Bi-LSTM deformation estimator (BLDE) to predict and compensate for the deformation of SBLLE based on the current force and torque data measured by force/torque (F/T) sensors. First, we choose four movements including squatting down and up, waist twisting, left foot lifting, and right foot lifting to mimic the constituent action of walking motion. The deformation data set is obtained through the motion capture analysis system and offline planning trajectories, and the relative F/T data set is obtained by the F/T sensors embedded in the feet of SBLLE. Second, the BiLSTM network is trained to learn the relationship between the deformation and F/T and verified on the test set. After that, BLDE is added to the control system of SBLLE to estimate and compensate for the deformation. Finally, four same movements and the walking experiment are conducted on the exoskeleton AutoLEE-G2 with BLDE. The experimental results have proven that BLDE can predict and compensate for deformation by only using F/T sensors.
{"title":"Estimation of Deformation for Self-balancing Lower Limb Exoskeleton Only Using Force/Torque Sensors","authors":"Ziqiang Chen, Ming Yang, Feng Li, Wentao Li, Jinke Li, Dingkui Tian, Jianquan Sun, Yong He, Xinyu Wu","doi":"10.1109/ROBIO58561.2023.10354999","DOIUrl":"https://doi.org/10.1109/ROBIO58561.2023.10354999","url":null,"abstract":"This paper presents a general estimation method of deformation for the self-balancing lower limb exoskeleton (SBLLE). In particular, we propose a Bi-LSTM deformation estimator (BLDE) to predict and compensate for the deformation of SBLLE based on the current force and torque data measured by force/torque (F/T) sensors. First, we choose four movements including squatting down and up, waist twisting, left foot lifting, and right foot lifting to mimic the constituent action of walking motion. The deformation data set is obtained through the motion capture analysis system and offline planning trajectories, and the relative F/T data set is obtained by the F/T sensors embedded in the feet of SBLLE. Second, the BiLSTM network is trained to learn the relationship between the deformation and F/T and verified on the test set. After that, BLDE is added to the control system of SBLLE to estimate and compensate for the deformation. Finally, four same movements and the walking experiment are conducted on the exoskeleton AutoLEE-G2 with BLDE. The experimental results have proven that BLDE can predict and compensate for deformation by only using F/T sensors.","PeriodicalId":505134,"journal":{"name":"2023 IEEE International Conference on Robotics and Biomimetics (ROBIO)","volume":"87 6","pages":"1-6"},"PeriodicalIF":0.0,"publicationDate":"2023-12-04","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"139186764","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
In this study, we develop an automatic control system to perform the reach-to-grasp movement of a 7-DOF (Degrees of Freedom) robotic arm that has the same DOFs as a human arm, and an end-effector with the same shape as a human hand. The 6-DOF pose of the object to be grasped is estimated in real time only from RGB images using a neural network based object pose estimation model. Based on this information, motion planning is performed to automatically control the reach-to-grasp movement of the robotic arm. In the evaluation experiment, the 7-DOF robotic arm performs reach-to-grasp movements for a household object in different poses using the developed control system. The results show that the control system developed in this study can automatically control the reach-to-grasp movement to an object in a certain arbitrary pose.
{"title":"Automatic Control System for Reach-to-Grasp Movement of a 7-DOF Robotic Arm Using Object Pose Estimation with an RGB Camera","authors":"Shuting Bai, Jiazhen Guo, Yinlai Jiang, Hiroshi Yokoi, Shunta Togo","doi":"10.1109/ROBIO58561.2023.10354531","DOIUrl":"https://doi.org/10.1109/ROBIO58561.2023.10354531","url":null,"abstract":"In this study, we develop an automatic control system to perform the reach-to-grasp movement of a 7-DOF (Degrees of Freedom) robotic arm that has the same DOFs as a human arm, and an end-effector with the same shape as a human hand. The 6-DOF pose of the object to be grasped is estimated in real time only from RGB images using a neural network based object pose estimation model. Based on this information, motion planning is performed to automatically control the reach-to-grasp movement of the robotic arm. In the evaluation experiment, the 7-DOF robotic arm performs reach-to-grasp movements for a household object in different poses using the developed control system. The results show that the control system developed in this study can automatically control the reach-to-grasp movement to an object in a certain arbitrary pose.","PeriodicalId":505134,"journal":{"name":"2023 IEEE International Conference on Robotics and Biomimetics (ROBIO)","volume":"13 2","pages":"1-6"},"PeriodicalIF":0.0,"publicationDate":"2023-12-04","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"139186772","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2023-12-04DOI: 10.1109/ROBIO58561.2023.10354624
Yinong Ye, Yongming Yue, Wei Gao, Shiwu Zhang
The walking control of bipedal robots poses challenges due to inherent coupling among the robot’s degrees of freedom. This paper introduces an approach to address this challenge by using decoupled control in the sagittal and frontal planes. The proposed control method takes advantage of Hybrid Zero Dynamics and Hybrid-Linear Inverted Pendulum for sagittal and frontal plane dynamics, respectively. The hybrid controller is successfully validated on a bipedal robot RobBIE, whose torso inertia is relatively high and if not adequately controlled can easily violate the point mass assumption in many reduced-order model based walking controllers developed previously. With the help of full-model based Hybrid Zero Dynamics, the robot can achieve stable walking behaviors at different velocities and adapt to various terrains and even moderate disturbances.
{"title":"Decoupled Control of Bipedal Locomotion Based on HZD and H-LIP","authors":"Yinong Ye, Yongming Yue, Wei Gao, Shiwu Zhang","doi":"10.1109/ROBIO58561.2023.10354624","DOIUrl":"https://doi.org/10.1109/ROBIO58561.2023.10354624","url":null,"abstract":"The walking control of bipedal robots poses challenges due to inherent coupling among the robot’s degrees of freedom. This paper introduces an approach to address this challenge by using decoupled control in the sagittal and frontal planes. The proposed control method takes advantage of Hybrid Zero Dynamics and Hybrid-Linear Inverted Pendulum for sagittal and frontal plane dynamics, respectively. The hybrid controller is successfully validated on a bipedal robot RobBIE, whose torso inertia is relatively high and if not adequately controlled can easily violate the point mass assumption in many reduced-order model based walking controllers developed previously. With the help of full-model based Hybrid Zero Dynamics, the robot can achieve stable walking behaviors at different velocities and adapt to various terrains and even moderate disturbances.","PeriodicalId":505134,"journal":{"name":"2023 IEEE International Conference on Robotics and Biomimetics (ROBIO)","volume":"109 4","pages":"1-6"},"PeriodicalIF":0.0,"publicationDate":"2023-12-04","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"139186778","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2023-12-04DOI: 10.1109/ROBIO58561.2023.10354748
Di Wu, Yuping Ye, Feifei Gu, Zhan Song
With the fast development of computer vision and artificial intelligence, many technologies from these fields have been introduced to the medical domain. Accurate estimation of burnt skin area is crucial for treatment plan selection and prognostic decision-making. However, state-of-art estimation of burnt skin area exhibits inadequate accuracy and acquisition efficiency. In this paper, a burnt skin acquisition system based on the infrared structured light 3D imaging method is developed. To accurately segment the burnt skin point cloud from the raw point cloud acquired by the proposed system, we employ the Segment Anything Model (SAM). Subsequently, the point clouds segmented from different views are registered using pre-calibrated parameters. Moreover, the surface reconstruction algorithm is employed to generate triangular meshes. Finally, we calculate the area of all the triangular mesh facets to represent the area of burnt skin. Several experiments were conducted to demonstrate the accuracy of the proposed method.
{"title":"A Quick Means for the Burnt Skin Area Calculation via Multiple-view Structured Light Sensors","authors":"Di Wu, Yuping Ye, Feifei Gu, Zhan Song","doi":"10.1109/ROBIO58561.2023.10354748","DOIUrl":"https://doi.org/10.1109/ROBIO58561.2023.10354748","url":null,"abstract":"With the fast development of computer vision and artificial intelligence, many technologies from these fields have been introduced to the medical domain. Accurate estimation of burnt skin area is crucial for treatment plan selection and prognostic decision-making. However, state-of-art estimation of burnt skin area exhibits inadequate accuracy and acquisition efficiency. In this paper, a burnt skin acquisition system based on the infrared structured light 3D imaging method is developed. To accurately segment the burnt skin point cloud from the raw point cloud acquired by the proposed system, we employ the Segment Anything Model (SAM). Subsequently, the point clouds segmented from different views are registered using pre-calibrated parameters. Moreover, the surface reconstruction algorithm is employed to generate triangular meshes. Finally, we calculate the area of all the triangular mesh facets to represent the area of burnt skin. Several experiments were conducted to demonstrate the accuracy of the proposed method.","PeriodicalId":505134,"journal":{"name":"2023 IEEE International Conference on Robotics and Biomimetics (ROBIO)","volume":"76 5","pages":"1-6"},"PeriodicalIF":0.0,"publicationDate":"2023-12-04","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"139186953","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2023-12-04DOI: 10.1109/ROBIO58561.2023.10354793
Jiangtao Luo, Dongbo Zhang, Tao Yi
As a representative bunch-type fruit,the collision-free and undamaged harvesting of grapes is of great significance. To obtain accurate 3D spatial semantic information,this paper proposes a method for multi-feature enhanced semantic segmentation model based on Mask R-CNN and PointNet++. Firstly, a depth camera is used to obtain RGBD images. The RGB images are then inputted into the Mask-RCNN network for fast detection of grape bunches. The color and depth information are fused and transformed into point cloud data, followed by the estimation of normal vectors. Finally, the nine-dimensional point cloud,which include spatial location, color information, and normal vectors, are inputted into the improved PointNet++ network to achieve semantic segmentation of grape bunches, peduncles, and leaves. This process obtains the extraction of spatial semantic information from the surrounding area of the bunches. The experimental results show that by incorporating normal vector and color features, the overall accuracy of point cloud segmentation increases to 93.7%, with a mean accuracy of 81.8%. This represents a significant improvement of 12.1% and 13.5% compared to using only positional features. The results demonstrate that the model method presented in this paper can effectively provide precise 3D semantic information to the robot while ensuring both speed and accuracy. This lays the groundwork for subsequent collision-free and damage-free picking.
{"title":"3D Semantic Segmentation for Grape Bunch Point Cloud Based on Feature Enhancement","authors":"Jiangtao Luo, Dongbo Zhang, Tao Yi","doi":"10.1109/ROBIO58561.2023.10354793","DOIUrl":"https://doi.org/10.1109/ROBIO58561.2023.10354793","url":null,"abstract":"As a representative bunch-type fruit,the collision-free and undamaged harvesting of grapes is of great significance. To obtain accurate 3D spatial semantic information,this paper proposes a method for multi-feature enhanced semantic segmentation model based on Mask R-CNN and PointNet++. Firstly, a depth camera is used to obtain RGBD images. The RGB images are then inputted into the Mask-RCNN network for fast detection of grape bunches. The color and depth information are fused and transformed into point cloud data, followed by the estimation of normal vectors. Finally, the nine-dimensional point cloud,which include spatial location, color information, and normal vectors, are inputted into the improved PointNet++ network to achieve semantic segmentation of grape bunches, peduncles, and leaves. This process obtains the extraction of spatial semantic information from the surrounding area of the bunches. The experimental results show that by incorporating normal vector and color features, the overall accuracy of point cloud segmentation increases to 93.7%, with a mean accuracy of 81.8%. This represents a significant improvement of 12.1% and 13.5% compared to using only positional features. The results demonstrate that the model method presented in this paper can effectively provide precise 3D semantic information to the robot while ensuring both speed and accuracy. This lays the groundwork for subsequent collision-free and damage-free picking.","PeriodicalId":505134,"journal":{"name":"2023 IEEE International Conference on Robotics and Biomimetics (ROBIO)","volume":"63 1","pages":"1-6"},"PeriodicalIF":0.0,"publicationDate":"2023-12-04","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"139187023","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2023-12-04DOI: 10.1109/ROBIO58561.2023.10354726
W. Ng, Han Yi Wang, Zheng Li
With the increasing surgical need in our aging society, there is a lack of experienced surgical assistants, such as scrub nurses. To facilitate the training of junior scrub nurses and to reduce human errors, e.g., missing surgical items, we develop a speech-image based multimodal AI framework to assist scrub nurses in the operating room. The proposed framework allows real-time instrument type identification and instance detection, which enables junior scrub nurses to become more familiar with the surgical instruments and guides them throughout the surgical procedure. We construct an ex-vivo video-assisted thorascopic surgery dataset and benchmark it on common object detection models, reaching an average precision of 98.5% and an average recall of 98.9% on the state-of-the-art YOLO-v7. Additionally, we implement an oriented bounding box version of YOLO-v7 to address the undesired bounding box suppression in instrument crossing over. By achieving an average precision of 95.6% and an average recall of 97.4%, we improve the average recall by up to 9.2% compared to the previous oriented bounding box version of YOLO-v5. To minimize distraction during surgery, we adopt a deep learning-based automatic speech recognition model to allow surgeons to concentrate on the procedure. Our physical demonstration substantiates the feasibility of the proposed framework in providing real-time guidance and assistance for scrub nurses.
{"title":"Speech-image based Multimodal AI Interaction for Scrub Nurse Assistance in the Operating Room","authors":"W. Ng, Han Yi Wang, Zheng Li","doi":"10.1109/ROBIO58561.2023.10354726","DOIUrl":"https://doi.org/10.1109/ROBIO58561.2023.10354726","url":null,"abstract":"With the increasing surgical need in our aging society, there is a lack of experienced surgical assistants, such as scrub nurses. To facilitate the training of junior scrub nurses and to reduce human errors, e.g., missing surgical items, we develop a speech-image based multimodal AI framework to assist scrub nurses in the operating room. The proposed framework allows real-time instrument type identification and instance detection, which enables junior scrub nurses to become more familiar with the surgical instruments and guides them throughout the surgical procedure. We construct an ex-vivo video-assisted thorascopic surgery dataset and benchmark it on common object detection models, reaching an average precision of 98.5% and an average recall of 98.9% on the state-of-the-art YOLO-v7. Additionally, we implement an oriented bounding box version of YOLO-v7 to address the undesired bounding box suppression in instrument crossing over. By achieving an average precision of 95.6% and an average recall of 97.4%, we improve the average recall by up to 9.2% compared to the previous oriented bounding box version of YOLO-v5. To minimize distraction during surgery, we adopt a deep learning-based automatic speech recognition model to allow surgeons to concentrate on the procedure. Our physical demonstration substantiates the feasibility of the proposed framework in providing real-time guidance and assistance for scrub nurses.","PeriodicalId":505134,"journal":{"name":"2023 IEEE International Conference on Robotics and Biomimetics (ROBIO)","volume":"73 2","pages":"1-6"},"PeriodicalIF":0.0,"publicationDate":"2023-12-04","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"139187027","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
京公网安备 11010802042870号
京ICP备2023020795号-1
扫码关注我们
求助内容：
应助结果提醒方式：