{"title":"延迟强化学习收敛于人类安静站姿的间歇控制","authors":"Yongkun Zhao , Balint K. Hodossy , Shibo Jing , Masahiro Todoh , Dario Farina","doi":"10.1016/j.medengphy.2024.104197","DOIUrl":null,"url":null,"abstract":"<div><p>The neural control of human quiet stance remains controversial, with classic views suggesting a limited role of the brain and recent findings conversely indicating direct cortical control of muscles during upright posture. Conceptual neural feedback control models have been proposed and tested against experimental evidence. The most renowned model is the continuous impedance control model. However, when time delays are included in this model to simulate neural transmission, the continuous controller becomes unstable. Another model, the intermittent control model, assumes that the central nervous system (CNS) activates muscles intermittently, and not continuously, to counteract gravitational torque. In this study, a delayed reinforcement learning algorithm was developed to seek optimal control policy to balance a one-segment inverted pendulum model representing the human body. According to this approach, there was no a-priori strategy imposed on the controller but rather the optimal strategy emerged from the reward-based learning. The simulation results indicated that the optimal neural controller exhibits intermittent, and not continuous, characteristics, in agreement with the possibility that the CNS intermittently provides neural feedback torque to maintain an upright posture.</p></div>","PeriodicalId":49836,"journal":{"name":"Medical Engineering & Physics","volume":"130 ","pages":"Article 104197"},"PeriodicalIF":1.7000,"publicationDate":"2024-08-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.sciencedirect.com/science/article/pii/S1350453324000985/pdfft?md5=f8324b603d1db550e3ab02402a64146c&pid=1-s2.0-S1350453324000985-main.pdf","citationCount":"0","resultStr":"{\"title\":\"Delayed reinforcement learning converges to intermittent control for human quiet stance\",\"authors\":\"Yongkun Zhao , Balint K. Hodossy , Shibo Jing , Masahiro Todoh , Dario Farina\",\"doi\":\"10.1016/j.medengphy.2024.104197\",\"DOIUrl\":null,\"url\":null,\"abstract\":\"<div><p>The neural control of human quiet stance remains controversial, with classic views suggesting a limited role of the brain and recent findings conversely indicating direct cortical control of muscles during upright posture. Conceptual neural feedback control models have been proposed and tested against experimental evidence. The most renowned model is the continuous impedance control model. However, when time delays are included in this model to simulate neural transmission, the continuous controller becomes unstable. Another model, the intermittent control model, assumes that the central nervous system (CNS) activates muscles intermittently, and not continuously, to counteract gravitational torque. In this study, a delayed reinforcement learning algorithm was developed to seek optimal control policy to balance a one-segment inverted pendulum model representing the human body. According to this approach, there was no a-priori strategy imposed on the controller but rather the optimal strategy emerged from the reward-based learning. The simulation results indicated that the optimal neural controller exhibits intermittent, and not continuous, characteristics, in agreement with the possibility that the CNS intermittently provides neural feedback torque to maintain an upright posture.</p></div>\",\"PeriodicalId\":49836,\"journal\":{\"name\":\"Medical Engineering & Physics\",\"volume\":\"130 \",\"pages\":\"Article 104197\"},\"PeriodicalIF\":1.7000,\"publicationDate\":\"2024-08-01\",\"publicationTypes\":\"Journal Article\",\"fieldsOfStudy\":null,\"isOpenAccess\":false,\"openAccessPdf\":\"https://www.sciencedirect.com/science/article/pii/S1350453324000985/pdfft?md5=f8324b603d1db550e3ab02402a64146c&pid=1-s2.0-S1350453324000985-main.pdf\",\"citationCount\":\"0\",\"resultStr\":null,\"platform\":\"Semanticscholar\",\"paperid\":null,\"PeriodicalName\":\"Medical Engineering & Physics\",\"FirstCategoryId\":\"5\",\"ListUrlMain\":\"https://www.sciencedirect.com/science/article/pii/S1350453324000985\",\"RegionNum\":4,\"RegionCategory\":\"医学\",\"ArticlePicture\":[],\"TitleCN\":null,\"AbstractTextCN\":null,\"PMCID\":null,\"EPubDate\":\"\",\"PubModel\":\"\",\"JCR\":\"Q3\",\"JCRName\":\"ENGINEERING, BIOMEDICAL\",\"Score\":null,\"Total\":0}","platform":"Semanticscholar","paperid":null,"PeriodicalName":"Medical Engineering & Physics","FirstCategoryId":"5","ListUrlMain":"https://www.sciencedirect.com/science/article/pii/S1350453324000985","RegionNum":4,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q3","JCRName":"ENGINEERING, BIOMEDICAL","Score":null,"Total":0}
Delayed reinforcement learning converges to intermittent control for human quiet stance
The neural control of human quiet stance remains controversial, with classic views suggesting a limited role of the brain and recent findings conversely indicating direct cortical control of muscles during upright posture. Conceptual neural feedback control models have been proposed and tested against experimental evidence. The most renowned model is the continuous impedance control model. However, when time delays are included in this model to simulate neural transmission, the continuous controller becomes unstable. Another model, the intermittent control model, assumes that the central nervous system (CNS) activates muscles intermittently, and not continuously, to counteract gravitational torque. In this study, a delayed reinforcement learning algorithm was developed to seek optimal control policy to balance a one-segment inverted pendulum model representing the human body. According to this approach, there was no a-priori strategy imposed on the controller but rather the optimal strategy emerged from the reward-based learning. The simulation results indicated that the optimal neural controller exhibits intermittent, and not continuous, characteristics, in agreement with the possibility that the CNS intermittently provides neural feedback torque to maintain an upright posture.
期刊介绍:
Medical Engineering & Physics provides a forum for the publication of the latest developments in biomedical engineering, and reflects the essential multidisciplinary nature of the subject. The journal publishes in-depth critical reviews, scientific papers and technical notes. Our focus encompasses the application of the basic principles of physics and engineering to the development of medical devices and technology, with the ultimate aim of producing improvements in the quality of health care.Topics covered include biomechanics, biomaterials, mechanobiology, rehabilitation engineering, biomedical signal processing and medical device development. Medical Engineering & Physics aims to keep both engineers and clinicians abreast of the latest applications of technology to health care.