Pub Date : 2024-06-26DOI: 10.1109/THMS.2024.3412273
Xi Fang;Hui Yang;Liu Shi;Yilong Wang;Li Li
With the widespread adoption of smartphones and mobile Internet, understanding user behavior and improving user experience are critical. This article introduces semantic-aware (SA)-BERT, a novel model that integrates spatio-temporal and semantic information to represent App usage effectively. Leveraging BERT, SA-BERT captures rich contextual information. By introducing a specific objective function to represent the cooccurrence of App-time-location paths, SA-BERT can effectively model complex App usage structures. Based on this method, we adopt the learned embedding vectors in App usage prediction tasks. We evaluate the performance of SA-BERT using a large-scale real-world dataset. As demonstrated in the numerous experimental results, our model outperformed other strategies evidently. In terms of the prediction accuracy, we achieve a performance gain of 34.9% compared with widely used the SA representation learning via graph convolutional network (SA-GCN), and 134.4% than the context-aware App usage prediction with heterogeneous graph embedding. In addition, we reduced 79.27% training time compared with SA-GCN.
{"title":"BERT-Based Semantic-Aware Heterogeneous Graph Embedding Method for Enhancing App Usage Prediction Accuracy","authors":"Xi Fang;Hui Yang;Liu Shi;Yilong Wang;Li Li","doi":"10.1109/THMS.2024.3412273","DOIUrl":"10.1109/THMS.2024.3412273","url":null,"abstract":"With the widespread adoption of smartphones and mobile Internet, understanding user behavior and improving user experience are critical. This article introduces semantic-aware (SA)-BERT, a novel model that integrates spatio-temporal and semantic information to represent App usage effectively. Leveraging BERT, SA-BERT captures rich contextual information. By introducing a specific objective function to represent the cooccurrence of App-time-location paths, SA-BERT can effectively model complex App usage structures. Based on this method, we adopt the learned embedding vectors in App usage prediction tasks. We evaluate the performance of SA-BERT using a large-scale real-world dataset. As demonstrated in the numerous experimental results, our model outperformed other strategies evidently. In terms of the prediction accuracy, we achieve a performance gain of 34.9% compared with widely used the SA representation learning via graph convolutional network (SA-GCN), and 134.4% than the context-aware App usage prediction with heterogeneous graph embedding. In addition, we reduced 79.27% training time compared with SA-GCN.","PeriodicalId":48916,"journal":{"name":"IEEE Transactions on Human-Machine Systems","volume":null,"pages":null},"PeriodicalIF":3.5,"publicationDate":"2024-06-26","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141500638","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-06-26DOI: 10.1109/THMS.2024.3412910
Shaodong Li;Peiyuan Gao;Yongzheng Chen
In robot-assisted live-line maintenance, bilateral teleoperation is still a popular and effective approach in assisting operators to accomplish hazards tasks. Particularly, teleoperation under overhead power lines attach greater expectation on safe operation and telepresence. In this article, we propose a visual-haptic bilateral teleoperation strategy, i.e., EMGP-VH , based on visual guidance, haptic constraint and mixed reality (MR) augmentation. To the best of our knowledge, electromagnetic field is first applied to serve the path planning of teleoperation in live-line maintenance. In visual guidance, EMG-potential fields are integrated into RRT* to calculate a low-energy path. At the same time, real-time haptic constraint is calculated based on a tube virtual fixture. MR augmentation also works as an indispensable part in both the platform construction and visual guidance. Our proposal has been extensively compared using seven objective performances and three subjective questionnaires both in simulation and real-world experiment with five different scenes and two approaches state-of-the-art, respectively. Functionality of EMGP-RRT* and effectiveness of haptic constraint are further analyzed. Results show that EMGP-RRT* has significant improvements both in searching efficiency and safety performances; and the proposed system ( EMGP-VH ) significantly contributes to improving telepresence and ensuring safe operations during live-line maintenance, resulting in a 30% reduction in operation time and a 60% decrease in trajectory offset.
{"title":"A Bilateral Teleoperation Strategy Augmented by EMGP-VH for Live-Line Maintenance Robot","authors":"Shaodong Li;Peiyuan Gao;Yongzheng Chen","doi":"10.1109/THMS.2024.3412910","DOIUrl":"10.1109/THMS.2024.3412910","url":null,"abstract":"In robot-assisted live-line maintenance, bilateral teleoperation is still a popular and effective approach in assisting operators to accomplish hazards tasks. Particularly, teleoperation under overhead power lines attach greater expectation on safe operation and telepresence. In this article, we propose a visual-haptic bilateral teleoperation strategy, i.e., \u0000<italic>EMGP-VH</i>\u0000, based on visual guidance, haptic constraint and mixed reality (MR) augmentation. To the best of our knowledge, electromagnetic field is first applied to serve the path planning of teleoperation in live-line maintenance. In visual guidance, EMG-potential fields are integrated into \u0000<italic>RRT*</i>\u0000 to calculate a low-energy path. At the same time, real-time haptic constraint is calculated based on a tube virtual fixture. MR augmentation also works as an indispensable part in both the platform construction and visual guidance. Our proposal has been extensively compared using seven objective performances and three subjective questionnaires both in simulation and real-world experiment with five different scenes and two approaches state-of-the-art, respectively. Functionality of \u0000<italic>EMGP-RRT*</i>\u0000 and effectiveness of haptic constraint are further analyzed. Results show that \u0000<italic>EMGP-RRT*</i>\u0000 has significant improvements both in searching efficiency and safety performances; and the proposed system (\u0000<italic>EMGP-VH</i>\u0000) significantly contributes to improving telepresence and ensuring safe operations during live-line maintenance, resulting in a 30% reduction in operation time and a 60% decrease in trajectory offset.","PeriodicalId":48916,"journal":{"name":"IEEE Transactions on Human-Machine Systems","volume":null,"pages":null},"PeriodicalIF":3.5,"publicationDate":"2024-06-26","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141500637","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-06-25DOI: 10.1109/THMS.2024.3408841
Umair Rehman;Shi Cao;Carolyn G. Macgregor
In this article, we used a computational cognitive architecture called queuing network–adaptive control of thought rational–situation awareness (QN–ACTR–SA) to model and simulate the brake perception response time (BPRT) to visual roadway hazards. The model incorporates an integrated driver model to simulate human driving behavior and uses a dynamic visual sampling model to simulate how drivers allocate their attention. We validated the model by comparing its results to empirical data from human participants who encountered on-road and roadside hazards in a simulated driving environment. The results showed that BPRT was shorter for on-road hazards compared to roadside hazards and that the overall model fitness had a mean absolute percentage error of 9.4% and a root mean squared error of 0.13 s. The modeling results demonstrated that QN–ACTR–SA could effectively simulate BPRT to both on-road and roadside hazards and capture the difference between the two contrasting conditions.
{"title":"Modeling Brake Perception Response Time in On-Road and Roadside Hazards Using an Integrated Cognitive Architecture","authors":"Umair Rehman;Shi Cao;Carolyn G. Macgregor","doi":"10.1109/THMS.2024.3408841","DOIUrl":"10.1109/THMS.2024.3408841","url":null,"abstract":"In this article, we used a computational cognitive architecture called queuing network–adaptive control of thought rational–situation awareness (QN–ACTR–SA) to model and simulate the brake perception response time (BPRT) to visual roadway hazards. The model incorporates an integrated driver model to simulate human driving behavior and uses a dynamic visual sampling model to simulate how drivers allocate their attention. We validated the model by comparing its results to empirical data from human participants who encountered on-road and roadside hazards in a simulated driving environment. The results showed that BPRT was shorter for on-road hazards compared to roadside hazards and that the overall model fitness had a mean absolute percentage error of 9.4% and a root mean squared error of 0.13 s. The modeling results demonstrated that QN–ACTR–SA could effectively simulate BPRT to both on-road and roadside hazards and capture the difference between the two contrasting conditions.","PeriodicalId":48916,"journal":{"name":"IEEE Transactions on Human-Machine Systems","volume":null,"pages":null},"PeriodicalIF":3.5,"publicationDate":"2024-06-25","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141516477","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Vision algorithms face challenges of limited visual presentation and unreliability in pedestrian activity assessment. In this article, we introduce LANDER, an interactive analysis system for visual exploration of pedestrian activity and uncertainty in surveillance videos. This visual analytics system focuses on three common categories of uncertainties in object tracking and action recognition. LANDER offers an overview visualization of activity and uncertainty, along with spatio-temporal exploration views closely associated with the scene. Expert evaluation and user study indicate that LANDER outperforms traditional video exploration in data presentation and analysis workflow. Specifically, compared to the baseline method, it excels in reducing retrieval time ( $p< $ 0.01), enhancing uncertainty identification ( $p< $ 0.05), and improving the user experience ( $p< $ 0.05).
{"title":"LANDER: Visual Analysis of Activity and Uncertainty in Surveillance Video","authors":"Tong Li;Guodao Sun;Baofeng Chang;Yunchao Wang;Qi Jiang;Yuanzhong Ying;Li Jiang;Haixia Wang;Ronghua Liang","doi":"10.1109/THMS.2024.3409722","DOIUrl":"10.1109/THMS.2024.3409722","url":null,"abstract":"Vision algorithms face challenges of limited visual presentation and unreliability in pedestrian activity assessment. In this article, we introduce LANDER, an interactive analysis system for visual exploration of pedestrian activity and uncertainty in surveillance videos. This visual analytics system focuses on three common categories of uncertainties in object tracking and action recognition. LANDER offers an overview visualization of activity and uncertainty, along with spatio-temporal exploration views closely associated with the scene. Expert evaluation and user study indicate that LANDER outperforms traditional video exploration in data presentation and analysis workflow. Specifically, compared to the baseline method, it excels in reducing retrieval time (\u0000<inline-formula><tex-math>$p< $</tex-math></inline-formula>\u0000 0.01), enhancing uncertainty identification (\u0000<inline-formula><tex-math>$p< $</tex-math></inline-formula>\u0000 0.05), and improving the user experience (\u0000<inline-formula><tex-math>$p< $</tex-math></inline-formula>\u0000 0.05).","PeriodicalId":48916,"journal":{"name":"IEEE Transactions on Human-Machine Systems","volume":null,"pages":null},"PeriodicalIF":3.5,"publicationDate":"2024-06-24","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141500636","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-06-18DOI: 10.1109/THMS.2024.3407333
Yahui Liu;Jingyuan Li;Yingbo Sun;Xuewu Ji;Chen Lv
Accurate and robust risk prediction on curved roads can significantly reduce lane departure accidents and improve traffic safety. However, limited study has considered dynamic driver-related factors in risk prediction, resulting in poor algorithm adaptiveness to individual differences. This article presents a novel personalized risk prediction method with consideration of driver turning behavior and workload by using the predicted vehicle trajectory.First, driving simulation experiments are conducted to collect synchronized trajectory data, vehicle dynamic data, and eye movement data. The drivers are distracted by answering questions via a Bluetooth headset, leading to an increased cognitive workload. Secondly, the k -means clustering algorithm is utilized to extract two turning behaviors: driving toward the inner and outer side of a curved road. The turning behavior of each trajectory is then recognized using the trajectory data. In addition, the driver workload is recognized using the vehicle dynamic features and eye movement features. Thirdly, an extra personalization index is introduced to a long short-term memory encoder–decoder trajectory prediction network. This index integrates the driver turning behavior and workload information. After introducing the personalization index, the root-mean-square errors of the proposed network are reduced by 15.6%, 23.5%, and 29.1% with prediction horizons of 2, 3, and 4 s, respectively. Fourthly, the risk potential field theory is employed for risk prediction using the predicted trajectory data. This approach implicitly incorporates the driver's personalized information into risk prediction.
{"title":"Personalized Trajectory-based Risk Prediction on Curved Roads with Consideration of Driver Turning Behavior and Workload","authors":"Yahui Liu;Jingyuan Li;Yingbo Sun;Xuewu Ji;Chen Lv","doi":"10.1109/THMS.2024.3407333","DOIUrl":"https://doi.org/10.1109/THMS.2024.3407333","url":null,"abstract":"Accurate and robust risk prediction on curved roads can significantly reduce lane departure accidents and improve traffic safety. However, limited study has considered dynamic driver-related factors in risk prediction, resulting in poor algorithm adaptiveness to individual differences. This article presents a novel personalized risk prediction method with consideration of driver turning behavior and workload by using the predicted vehicle trajectory.First, driving simulation experiments are conducted to collect synchronized trajectory data, vehicle dynamic data, and eye movement data. The drivers are distracted by answering questions via a Bluetooth headset, leading to an increased cognitive workload. Secondly, the \u0000<italic>k</i>\u0000-means clustering algorithm is utilized to extract two turning behaviors: driving toward the inner and outer side of a curved road. The turning behavior of each trajectory is then recognized using the trajectory data. In addition, the driver workload is recognized using the vehicle dynamic features and eye movement features. Thirdly, an extra personalization index is introduced to a long short-term memory encoder–decoder trajectory prediction network. This index integrates the driver turning behavior and workload information. After introducing the personalization index, the root-mean-square errors of the proposed network are reduced by 15.6%, 23.5%, and 29.1% with prediction horizons of 2, 3, and 4 s, respectively. Fourthly, the risk potential field theory is employed for risk prediction using the predicted trajectory data. This approach implicitly incorporates the driver's personalized information into risk prediction.","PeriodicalId":48916,"journal":{"name":"IEEE Transactions on Human-Machine Systems","volume":null,"pages":null},"PeriodicalIF":3.5,"publicationDate":"2024-06-18","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141725650","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-06-17DOI: 10.1109/THMS.2024.3398631
Xiao-Xiao Zhang;Huai-Ning Wu;Jin-Liang Wang
In this article, the distributed formation control problem for a class of human-in-the-loop (HiTL) multiagent systems (MASs) is studied. A hidden Markov jump MAS is employed to model the HiTL MAS, which integrates the human models, the MAS model, and their interactions. The HiTL MAS investigated in this article is composed of two parts: a leader without human in the control loop and a group of followers in which each follower is simultaneously controlled by a human operator and an automation. For each follower, a hidden Markov model is used for modeling the human behaviors in consideration of the random nature of human internal state (HIS) reasoning and the uncertainty from HIS observation. By means of a stochastic Lyapunov function, a necessary and sufficient condition is first developed in terms of the linear matrix inequalities (LMIs) to ensure the formation of the HiTL MAS in the mean-square sense. Then, an LMI approach to the human-assistance control design is proposed for the automations in the followers to guarantee the mean-square formation of the HiTL MAS. Finally, simulation results are presented to verify the effectiveness of the proposed methods.
本文研究了一类人在回路（HiTL）多代理系统（MAS）的分布式编队控制问题。本文采用隐马尔可夫跃迁 MAS 对 HiTL MAS 进行建模，该模型集成了人类模型、MAS 模型以及它们之间的相互作用。本文研究的 HiTL MAS 由两部分组成：一个在控制环中没有人类的领导者和一组追随者，其中每个追随者同时受人类操作员和自动化设备的控制。考虑到人类内部状态（HIS）推理的随机性和 HIS 观察的不确定性，本文采用隐马尔可夫模型为每个追随者的人类行为建模。通过随机 Lyapunov 函数，首先从线性矩阵不等式（LMI）的角度提出了一个必要条件和充分条件，以确保在均方意义上形成 HiTL MAS。然后，为保证 HiTL MAS 的均方形成，对跟随者中的自动装置提出了一种线性矩阵不等式的人工辅助控制设计方法。最后，介绍了仿真结果，以验证所提方法的有效性。
{"title":"Distributed Formation Control for a Class of Human-in-the-Loop Multiagent Systems","authors":"Xiao-Xiao Zhang;Huai-Ning Wu;Jin-Liang Wang","doi":"10.1109/THMS.2024.3398631","DOIUrl":"https://doi.org/10.1109/THMS.2024.3398631","url":null,"abstract":"In this article, the distributed formation control problem for a class of human-in-the-loop (HiTL) multiagent systems (MASs) is studied. A hidden Markov jump MAS is employed to model the HiTL MAS, which integrates the human models, the MAS model, and their interactions. The HiTL MAS investigated in this article is composed of two parts: a leader without human in the control loop and a group of followers in which each follower is simultaneously controlled by a human operator and an automation. For each follower, a hidden Markov model is used for modeling the human behaviors in consideration of the random nature of human internal state (HIS) reasoning and the uncertainty from HIS observation. By means of a stochastic Lyapunov function, a necessary and sufficient condition is first developed in terms of the linear matrix inequalities (LMIs) to ensure the formation of the HiTL MAS in the mean-square sense. Then, an LMI approach to the human-assistance control design is proposed for the automations in the followers to guarantee the mean-square formation of the HiTL MAS. Finally, simulation results are presented to verify the effectiveness of the proposed methods.","PeriodicalId":48916,"journal":{"name":"IEEE Transactions on Human-Machine Systems","volume":null,"pages":null},"PeriodicalIF":3.5,"publicationDate":"2024-06-17","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141725599","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-06-06DOI: 10.1109/THMS.2024.3404101
Chana Chansri;Jakkree Srinonchat
The current sensor systems using the human–computer interface to develop a hand gesture recognition system remain challenging. This research presents the development of hand gesture recognition with 16-DoF glove sensors combined with a convolution neural network. The flex sensors are attached to 16 pivot joints of the human hand on the glove so that each knuckle flex can be measured while holding the object. The 16-DoF point sensors collecting circuit and adjustable buffer circuit were developed in this research to work with the Arduino Nano microcontroller to record each sensor's signal. This article investigates the time-series data of the flex sensor signal into 2-D colored images, concatenating the signals into one bigger image with a Gramian angular field and then recognition through a deep convolutional neural network (DCNN). The 16-DoF glove sensors were proposed for testing with three experiments using 8 models of DCNN recognition. These were conducted on 20 hand gesture recognition, 12 hand sign recognition, and object manipulation according to shape. The experimental results indicated that the best performance for the hand grasp experiment is 99.49% with Resnet 101, the hand sign experiment is 100% with Alexnet, and the object attribute experiment is 99.77% with InceptionNet V3.
{"title":"Utilizing Gramian Angular Fields and Convolution Neural Networks in Flex Sensors Glove for Human–Computer Interaction","authors":"Chana Chansri;Jakkree Srinonchat","doi":"10.1109/THMS.2024.3404101","DOIUrl":"https://doi.org/10.1109/THMS.2024.3404101","url":null,"abstract":"The current sensor systems using the human–computer interface to develop a hand gesture recognition system remain challenging. This research presents the development of hand gesture recognition with 16-DoF glove sensors combined with a convolution neural network. The flex sensors are attached to 16 pivot joints of the human hand on the glove so that each knuckle flex can be measured while holding the object. The 16-DoF point sensors collecting circuit and adjustable buffer circuit were developed in this research to work with the Arduino Nano microcontroller to record each sensor's signal. This article investigates the time-series data of the flex sensor signal into 2-D colored images, concatenating the signals into one bigger image with a Gramian angular field and then recognition through a deep convolutional neural network (DCNN). The 16-DoF glove sensors were proposed for testing with three experiments using 8 models of DCNN recognition. These were conducted on 20 hand gesture recognition, 12 hand sign recognition, and object manipulation according to shape. The experimental results indicated that the best performance for the hand grasp experiment is 99.49% with Resnet 101, the hand sign experiment is 100% with Alexnet, and the object attribute experiment is 99.77% with InceptionNet V3.","PeriodicalId":48916,"journal":{"name":"IEEE Transactions on Human-Machine Systems","volume":null,"pages":null},"PeriodicalIF":3.5,"publicationDate":"2024-06-06","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141725620","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-06-06DOI: 10.1109/THMS.2024.3407109
Syed T. Mubarrat;Antonio Fernandes;Suman K. Chowdhury
Recent advancements in virtual reality (VR) technology facilitate tracking real-world objects and users' movements in the virtual environment (VE) and inspire researchers to develop a physics-based haptic system (i.e., real object haptics) instead of computer-generated haptic feedback. However, there is limited research on the efficacy of such VR systems in enhancing operators’ sensorimotor learning for tasks that require high motor and physical demands. Therefore, this study aimed to design and evaluate the efficacy of a physics-based VR system that provides users with realistic cutaneous and kinesthetic haptic feedback. We designed a physics-based VR system, named PhyVirtual, and simulated human–robot collaborative (HRC) sequential pick-and-place lifting tasks in the VE. Participants performed the same tasks in the real environment (RE) with human–human collaboration instead of human–robot collaboration. We used a custom-designed questionnaire, the NASA-TLX, and electromyography activities from biceps, middle, and anterior deltoid muscles to determine user experience, workload, and neuromuscular dynamics, respectively. Overall, the majority of responses (>65%) demonstrated that the system is easy-to-use, easy-to-learn, and effective in improving motor skill performance. While compared to tasks performed in the RE, no significant difference was observed in the overall workload for the PhyVirtual system. The electromyography data exhibited similar trends ( p > 0.05; r > 0.89) for both environments. These results show that the PhyVirtual system is an effective tool to simulate safe human–robot collaboration commonly seen in many modern warehousing settings. Moreover, it can be used as a viable replacement for live sensorimotor training in a wide range of fields.
虚拟现实（VR）技术的最新进展有助于在虚拟环境（VE）中追踪真实世界中的物体和用户的动作，并激励研究人员开发基于物理的触觉系统（即真实物体触觉），而不是计算机生成的触觉反馈。然而，对于需要高运动和高体能要求的任务，此类 VR 系统在增强操作员感知运动学习方面的功效，目前的研究还很有限。因此，本研究旨在设计和评估基于物理的 VR 系统的功效，该系统可为用户提供逼真的皮肤和运动触觉反馈。我们设计了一个名为 PhyVirtual 的基于物理的 VR 系统，并在 VE 中模拟了人机协作（HRC）的顺序拾放搬运任务。参与者在真实环境（RE）中通过人与人之间的协作而不是人与机器人之间的协作来完成同样的任务。我们使用定制的调查问卷、NASA-TLX 以及肱二头肌、中肌和三角肌前肌的肌电图活动来分别确定用户体验、工作量和神经肌肉动态。总体而言，大多数反馈（>65%）表明该系统易于使用、易于学习，并能有效提高运动技能表现。与在 RE 中执行的任务相比，PhyVirtual 系统的总体工作量没有明显差异。两种环境下的肌电图数据呈现出相似的趋势（p > 0.05；r > 0.89）。这些结果表明，PhyVirtual 系统是模拟现代仓储环境中常见的安全人机协作的有效工具。此外，它还可以在广泛的领域替代现场传感器运动训练。
{"title":"A Physics-Based Virtual Reality Haptic System Design and Evaluation by Simulating Human-Robot Collaboration","authors":"Syed T. Mubarrat;Antonio Fernandes;Suman K. Chowdhury","doi":"10.1109/THMS.2024.3407109","DOIUrl":"https://doi.org/10.1109/THMS.2024.3407109","url":null,"abstract":"Recent advancements in virtual reality (VR) technology facilitate tracking real-world objects and users' movements in the virtual environment (VE) and inspire researchers to develop a physics-based haptic system (i.e., real object haptics) instead of computer-generated haptic feedback. However, there is limited research on the efficacy of such VR systems in enhancing operators’ sensorimotor learning for tasks that require high motor and physical demands. Therefore, this study aimed to design and evaluate the efficacy of a physics-based VR system that provides users with realistic cutaneous and kinesthetic haptic feedback. We designed a physics-based VR system, named PhyVirtual, and simulated human–robot collaborative (HRC) sequential pick-and-place lifting tasks in the VE. Participants performed the same tasks in the real environment (RE) with human–human collaboration instead of human–robot collaboration. We used a custom-designed questionnaire, the NASA-TLX, and electromyography activities from biceps, middle, and anterior deltoid muscles to determine user experience, workload, and neuromuscular dynamics, respectively. Overall, the majority of responses (>65%) demonstrated that the system is easy-to-use, easy-to-learn, and effective in improving motor skill performance. While compared to tasks performed in the RE, no significant difference was observed in the overall workload for the PhyVirtual system. The electromyography data exhibited similar trends (\u0000<italic>p</i>\u0000 > 0.05; \u0000<italic>r</i>\u0000 > 0.89) for both environments. These results show that the PhyVirtual system is an effective tool to simulate safe human–robot collaboration commonly seen in many modern warehousing settings. Moreover, it can be used as a viable replacement for live sensorimotor training in a wide range of fields.","PeriodicalId":48916,"journal":{"name":"IEEE Transactions on Human-Machine Systems","volume":null,"pages":null},"PeriodicalIF":3.5,"publicationDate":"2024-06-06","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141725598","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-06-05DOI: 10.1109/THMS.2024.3400290
M. S. Subodh Raj;Sudhish N. George
In this article, we propose a robust algorithm for the fast recovery of human actions from corrupted 3-D motion capture (mocap) sequences. The proposed algorithm can deal with misrepresentations and incomplete representations in mocap data simultaneously. Fast convergence of the proposed algorithm is ensured by minimizing the overhead associated with time and resource utilization. To this end, we have used an approximate singular value decomposition (SVD) based on QR decomposition and $ell _{2,1}$ norm minimization as a replacement for the conventional nuclear norm-based SVD. In addition, the proposed method is braced by incorporating the spatio-temporal properties of human action in the optimization problem. For this, we have introduced pair-wise hierarchical constraint and the trajectory movement constraint in the problem formulation. Finally, the proposed method is void of the requirement of a sizeable database for training the model. The algorithm can easily be adapted to work on any form of corrupted mocap sequences. The proposed algorithm is faster by 30% on average compared with the counterparts employing similar kinds of constraints with improved performance in recovery.
{"title":"A Fast and Efficient Approach for Human Action Recovery From Corrupted 3-D Motion Capture Data Using QR Decomposition-Based Approximate SVD","authors":"M. S. Subodh Raj;Sudhish N. George","doi":"10.1109/THMS.2024.3400290","DOIUrl":"https://doi.org/10.1109/THMS.2024.3400290","url":null,"abstract":"In this article, we propose a robust algorithm for the fast recovery of human actions from corrupted 3-D motion capture (mocap) sequences. The proposed algorithm can deal with misrepresentations and incomplete representations in mocap data simultaneously. Fast convergence of the proposed algorithm is ensured by minimizing the overhead associated with time and resource utilization. To this end, we have used an approximate singular value decomposition (SVD) based on QR decomposition and \u0000<inline-formula><tex-math>$ell _{2,1}$</tex-math></inline-formula>\u0000 norm minimization as a replacement for the conventional nuclear norm-based SVD. In addition, the proposed method is braced by incorporating the spatio-temporal properties of human action in the optimization problem. For this, we have introduced pair-wise hierarchical constraint and the trajectory movement constraint in the problem formulation. Finally, the proposed method is void of the requirement of a sizeable database for training the model. The algorithm can easily be adapted to work on any form of corrupted mocap sequences. The proposed algorithm is faster by 30% on average compared with the counterparts employing similar kinds of constraints with improved performance in recovery.","PeriodicalId":48916,"journal":{"name":"IEEE Transactions on Human-Machine Systems","volume":null,"pages":null},"PeriodicalIF":3.5,"publicationDate":"2024-06-05","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141725575","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-04-29DOI: 10.1109/THMS.2024.3385124
Biao Jin;Xiao Ma;Bojun Hu;Zhenkai Zhang;Zhuxian Lian;Biao Wang
Dynamic gesture recognition using millimeter-wave radar is a promising contactless mode of human–computer interaction with wide-ranging applications in various fields, such as intelligent homes, automatic driving, and sign language translation. However, the existing models have too many parameters and are unsuitable for embedded devices. To address this issue, we propose a dynamic gesture recognition method (named “Gesture-mmWAVE”) using millimeter-wave radar based on the multilevel feature fusion (MLFF) and transformer model. We first arrange each frame of the original echo collected by the frequency-modulated continuously modulated millimeter-wave radar in the Chirps × Samples format. Then, we use a 2-D fast Fourier transform to obtain the range-time map and Doppler-time map of gestures while improving the echo signal-to-noise ratio by coherent accumulation. Furthermore, we build an MLFF-transformer network for dynamic gesture recognition. The MLFF-transformer network comprises an MLFF module and a transformer module. The MLFF module employs the residual strategies to fuse the shallow, middle, and deep features and reduce the parameter size of the model using depthwise-separable convolution. The transformer module captures the global features of dynamic gestures and focuses on essential features using the multihead attention mechanism. The experimental results demonstrate that our proposed model achieves an average recognition accuracy of 99.11% on a dataset with 10% random interference. The scale of the proposed model is only 0.42M, which is 25% of that of the MobileNet V3-samll model. Thus, this method has excellent potential for application in embedded devices due to its small parameter size and high recognition accuracy.
{"title":"Gesture-mmWAVE: Compact and Accurate Millimeter-Wave Radar-Based Dynamic Gesture Recognition for Embedded Devices","authors":"Biao Jin;Xiao Ma;Bojun Hu;Zhenkai Zhang;Zhuxian Lian;Biao Wang","doi":"10.1109/THMS.2024.3385124","DOIUrl":"10.1109/THMS.2024.3385124","url":null,"abstract":"Dynamic gesture recognition using millimeter-wave radar is a promising contactless mode of human–computer interaction with wide-ranging applications in various fields, such as intelligent homes, automatic driving, and sign language translation. However, the existing models have too many parameters and are unsuitable for embedded devices. To address this issue, we propose a dynamic gesture recognition method (named “Gesture-mmWAVE”) using millimeter-wave radar based on the multilevel feature fusion (MLFF) and transformer model. We first arrange each frame of the original echo collected by the frequency-modulated continuously modulated millimeter-wave radar in the Chirps × Samples format. Then, we use a 2-D fast Fourier transform to obtain the range-time map and Doppler-time map of gestures while improving the echo signal-to-noise ratio by coherent accumulation. Furthermore, we build an MLFF-transformer network for dynamic gesture recognition. The MLFF-transformer network comprises an MLFF module and a transformer module. The MLFF module employs the residual strategies to fuse the shallow, middle, and deep features and reduce the parameter size of the model using depthwise-separable convolution. The transformer module captures the global features of dynamic gestures and focuses on essential features using the multihead attention mechanism. The experimental results demonstrate that our proposed model achieves an average recognition accuracy of 99.11% on a dataset with 10% random interference. The scale of the proposed model is only 0.42M, which is 25% of that of the MobileNet V3-samll model. Thus, this method has excellent potential for application in embedded devices due to its small parameter size and high recognition accuracy.","PeriodicalId":48916,"journal":{"name":"IEEE Transactions on Human-Machine Systems","volume":null,"pages":null},"PeriodicalIF":3.6,"publicationDate":"2024-04-29","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"140837281","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
京公网安备 11010802042870号
京ICP备2023020795号-1
扫码关注我们
求助内容：
应助结果提醒方式：