{"title":"TAD-Net: An approach for real-time action detection based on temporal convolution network and graph convolution network in digital twin shop-floor","authors":"Qing Hong, Yifeng Sun, Tingyu Liu, Liang Fu, Yunfeng Xie","doi":"10.12688/digitaltwin.17408.1","DOIUrl":null,"url":null,"abstract":"Background: Intelligent monitoring of human action in production is an important step to help standardize production processes and construct a digital twin shop-floor rapidly. Human action has a significant impact on the production safety and efficiency of a shop-floor, however, because of the high individual initiative of humans, it is difficult to realize real-time action detection in a digital twin shop-floor. Methods: We proposed a real-time detection approach for shop-floor production action. This approach used the sequence data of continuous human skeleton joints sequences as the input. We then reconstructed the Joint Classification-Regression Recurrent Neural Networks (JCR-RNN) based on Temporal Convolution Network (TCN) and Graph Convolution Network (GCN). We called this approach the Temporal Action Detection Net (TAD-Net), which realized real-time shop-floor production action detection. Results: The results of the verification experiment showed that our approach has achieved a high temporal positioning score, recognition speed, and accuracy when applied to the existing Online Action Detection (OAD) dataset and the Nanjing University of Science and Technology 3 Dimensions (NJUST3D) dataset. TAD-Net can meet the actual needs of the digital twin shop-floor. Conclusions: Our method has higher recognition accuracy, temporal positioning accuracy, and faster running speed than other mainstream network models, it can better meet actual application requirements, and has important research value and practical significance for standardizing shop-floor production processes, reducing production security risks, and contributing to the understanding of real-time production action.","PeriodicalId":29831,"journal":{"name":"Digital Twin","volume":null,"pages":null},"PeriodicalIF":0.0000,"publicationDate":"2021-12-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"5","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Digital Twin","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/10.12688/digitaltwin.17408.1","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"","JCRName":"","Score":null,"Total":0}
引用次数: 5
Abstract
Background: Intelligent monitoring of human action in production is an important step to help standardize production processes and construct a digital twin shop-floor rapidly. Human action has a significant impact on the production safety and efficiency of a shop-floor, however, because of the high individual initiative of humans, it is difficult to realize real-time action detection in a digital twin shop-floor. Methods: We proposed a real-time detection approach for shop-floor production action. This approach used the sequence data of continuous human skeleton joints sequences as the input. We then reconstructed the Joint Classification-Regression Recurrent Neural Networks (JCR-RNN) based on Temporal Convolution Network (TCN) and Graph Convolution Network (GCN). We called this approach the Temporal Action Detection Net (TAD-Net), which realized real-time shop-floor production action detection. Results: The results of the verification experiment showed that our approach has achieved a high temporal positioning score, recognition speed, and accuracy when applied to the existing Online Action Detection (OAD) dataset and the Nanjing University of Science and Technology 3 Dimensions (NJUST3D) dataset. TAD-Net can meet the actual needs of the digital twin shop-floor. Conclusions: Our method has higher recognition accuracy, temporal positioning accuracy, and faster running speed than other mainstream network models, it can better meet actual application requirements, and has important research value and practical significance for standardizing shop-floor production processes, reducing production security risks, and contributing to the understanding of real-time production action.
期刊介绍:
Digital Twin is a rapid multidisciplinary open access publishing platform for state-of-the-art, basic, scientific and applied research on digital twin technologies. Digital Twin covers all areas related digital twin technologies, including broad fields such as smart manufacturing, civil and industrial engineering, healthcare, agriculture, and many others. The platform is open to submissions from researchers, practitioners and experts, and all articles will benefit from open peer review.
The aim of Digital Twin is to advance the state-of-the-art in digital twin research and encourage innovation by highlighting efficient, robust and sustainable multidisciplinary applications across a variety of fields. Challenges can be addressed using theoretical, methodological, and technological approaches.
The scope of Digital Twin includes, but is not limited to, the following areas:
● Digital twin concepts, architecture, and frameworks
● Digital twin theory and method
● Digital twin key technologies and tools
● Digital twin applications and case studies
● Digital twin implementation
● Digital twin services
● Digital twin security
● Digital twin standards
Digital twin also focuses on applications within and across broad sectors including:
● Smart manufacturing
● Aviation and aerospace
● Smart cities and construction
● Healthcare and medicine
● Robotics
● Shipping, vehicles and railways
● Industrial engineering and engineering management
● Agriculture
● Mining
● Power, energy and environment
Digital Twin features a range of article types including research articles, case studies, method articles, study protocols, software tools, systematic reviews, data notes, brief reports, and opinion articles.