Pub Date : 2024-01-01DOI: 10.1109/OJCSYS.2023.3348704
Madeleine Shuhn-Tsuan Yuh;Kendric Ray Ortiz;Kylie Sue Sommer-Kohrt;Meeko Oishi;Neera Jain
Adaptive automation, automation which is responsive to the human's performance via the alteration of control laws or level of assistance, is an important tool for training humans to attain new skills when operating dynamical systems. When coupled with cognitive feedback, adaptive automation has the potential to further facilitate human training, but requires precise assessments of human progression through various learning stages. This is challenging because of the underlying dynamics, as well as the stochasticity inherent to human action. We propose a data-driven approach to assess learning stages in a complex quadrotor landing task that is responsive to stochastic, human-in-the-loop quadrotor dynamics. We represent each learning stage as a distribution of canonical trajectories for that learning stage, then employ kernel distribution embeddings in combination with a rule-based heuristic, to determine which canonical distribution a sample landing trajectory is closest to. We demonstrate our approach on experimental human subject data, and use our approach to evaluate the efficacy of cognitively-based adaptive automation designed to calibrate self-confidence. Our approach is more accurate than standard classification methods, such as nearest centroid assignment, which rely on metrics that are not inherently suited to analysis of trajectories of stochastic dynamical systems.
{"title":"Classification of Human Learning Stages via Kernel Distribution Embeddings","authors":"Madeleine Shuhn-Tsuan Yuh;Kendric Ray Ortiz;Kylie Sue Sommer-Kohrt;Meeko Oishi;Neera Jain","doi":"10.1109/OJCSYS.2023.3348704","DOIUrl":"https://doi.org/10.1109/OJCSYS.2023.3348704","url":null,"abstract":"Adaptive automation, automation which is responsive to the human's performance via the alteration of control laws or level of assistance, is an important tool for training humans to attain new skills when operating dynamical systems. When coupled with cognitive feedback, adaptive automation has the potential to further facilitate human training, but requires precise assessments of human progression through various learning stages. This is challenging because of the underlying dynamics, as well as the stochasticity inherent to human action. We propose a data-driven approach to assess learning stages in a complex quadrotor landing task that is responsive to stochastic, human-in-the-loop quadrotor dynamics. We represent each learning stage as a distribution of canonical trajectories for that learning stage, then employ kernel distribution embeddings in combination with a rule-based heuristic, to determine which canonical distribution a sample landing trajectory is closest to. We demonstrate our approach on experimental human subject data, and use our approach to evaluate the efficacy of cognitively-based adaptive automation designed to calibrate self-confidence. Our approach is more accurate than standard classification methods, such as nearest centroid assignment, which rely on metrics that are not inherently suited to analysis of trajectories of stochastic dynamical systems.","PeriodicalId":73299,"journal":{"name":"IEEE open journal of control systems","volume":"3 ","pages":"102-117"},"PeriodicalIF":0.0,"publicationDate":"2024-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=10378711","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"139976167","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2023-12-21DOI: 10.1109/OJCSYS.2023.3345772
{"title":"2023 Index IEEE Open Journal of Control Systems Vol.2","authors":"","doi":"10.1109/OJCSYS.2023.3345772","DOIUrl":"https://doi.org/10.1109/OJCSYS.2023.3345772","url":null,"abstract":"","PeriodicalId":73299,"journal":{"name":"IEEE open journal of control systems","volume":"2 ","pages":"477-483"},"PeriodicalIF":0.0,"publicationDate":"2023-12-21","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=10368194","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"139034097","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Trust model is a topic that first gained interest in organizational studies and then human factors in automation. Thanks to recent advances in human-robot interaction (HRI) and human-autonomy teaming, human trust in robots has gained growing interest among researchers and practitioners. This article focuses on a survey of computational models of human-robot trust and their applications in robotics and robot controls. The motivation is to provide an overview of the state-of-the-art computational methods to quantify trust so as to provide feedback and situational awareness in HRI. Different from other existing survey papers on human-robot trust models, we seek to provide in-depth coverage of the trust model categorization, formulation, and analysis, with a focus on their utilization in robotics and robot controls. The paper starts with a discussion of the difference between human-robot trust with general agent-agent trust, interpersonal trust, and human trust in automation and machines. A list of impacting factors for human-robot trust and different trust measurement approaches, and their corresponding scales are summarized. We then review existing computational human-robot trust models and discuss the pros and cons of each category of models. These include performance-centric algebraic, time-series, Markov decision process (MDP)/Partially Observable MDP (POMDP)-based, Gaussian-based, and dynamic Bayesian network (DBN)-based trust models. Following the summary of each computational human-robot trust model, we examine its utilization in robot control applications, if any. We also enumerate the main limitations and open questions in this field and discuss potential future research directions.
{"title":"Human Trust in Robots: A Survey on Trust Models and Their Controls/Robotics Applications","authors":"Yue Wang;Fangjian Li;Huanfei Zheng;Longsheng Jiang;Maziar Fooladi Mahani;Zhanrui Liao","doi":"10.1109/OJCSYS.2023.3345090","DOIUrl":"https://doi.org/10.1109/OJCSYS.2023.3345090","url":null,"abstract":"Trust model is a topic that first gained interest in organizational studies and then human factors in automation. Thanks to recent advances in human-robot interaction (HRI) and human-autonomy teaming, human trust in robots has gained growing interest among researchers and practitioners. This article focuses on a survey of computational models of human-robot trust and their applications in robotics and robot controls. The motivation is to provide an overview of the state-of-the-art computational methods to quantify trust so as to provide feedback and situational awareness in HRI. Different from other existing survey papers on human-robot trust models, we seek to provide in-depth coverage of the trust model categorization, formulation, and analysis, with a focus on their utilization in robotics and robot controls. The paper starts with a discussion of the difference between human-robot trust with general agent-agent trust, interpersonal trust, and human trust in automation and machines. A list of impacting factors for human-robot trust and different trust measurement approaches, and their corresponding scales are summarized. We then review existing computational human-robot trust models and discuss the pros and cons of each category of models. These include performance-centric algebraic, time-series, Markov decision process (MDP)/Partially Observable MDP (POMDP)-based, Gaussian-based, and dynamic Bayesian network (DBN)-based trust models. Following the summary of each computational human-robot trust model, we examine its utilization in robot control applications, if any. We also enumerate the main limitations and open questions in this field and discuss potential future research directions.","PeriodicalId":73299,"journal":{"name":"IEEE open journal of control systems","volume":"3 ","pages":"58-86"},"PeriodicalIF":0.0,"publicationDate":"2023-12-20","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=10366819","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"139431049","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2023-12-15DOI: 10.1109/OJCSYS.2023.3315635
{"title":"IEEE Control Systems Society Information","authors":"","doi":"10.1109/OJCSYS.2023.3315635","DOIUrl":"https://doi.org/10.1109/OJCSYS.2023.3315635","url":null,"abstract":"","PeriodicalId":73299,"journal":{"name":"IEEE open journal of control systems","volume":"2 ","pages":"C3-C3"},"PeriodicalIF":0.0,"publicationDate":"2023-12-15","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=10361563","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"138678663","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2023-12-15DOI: 10.1109/OJCSYS.2023.3315631
{"title":"IEEE Open Journal of Control Systems Publication Information","authors":"","doi":"10.1109/OJCSYS.2023.3315631","DOIUrl":"https://doi.org/10.1109/OJCSYS.2023.3315631","url":null,"abstract":"","PeriodicalId":73299,"journal":{"name":"IEEE open journal of control systems","volume":"2 ","pages":"C2-C2"},"PeriodicalIF":0.0,"publicationDate":"2023-12-15","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=10361595","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"138713667","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2023-12-15DOI: 10.1109/OJCSYS.2023.3343598
Takeshi Hatanaka;Takahiro Mochizuki;Takumi Sumino;José M. Maestre;Nikhil Chopra
In this article, we study a one-human-multiple-robot interaction for human-enabled multi-robot navigation in three dimensions. We employ two fully distributed control architectures designed based on human passivity and human passivity shortage. The first half of this article focuses on human modeling and analysis for the passivity-based control architecture through human operation data on a 3-D human-in-the-loop simulator. Specifically, we compare virtual reality (VR) interfaces with a traditional interface, and examine the impacts that VR technology has on human properties in terms of model accuracy, performance, passivity and workload, demonstrating that VR interfaces have a positive effect on all aspects. In contrast to 1-D operation, we confirm that operators hardly attain passivity regardless of the network structure, even with the VR interfaces. We thus take the passivity-shortage-based control architecture and analyze the degree of passivity shortage. We then observe through user studies that operators tend to meet the degree of shortage needed to prove closed-loop stability.
{"title":"Human Modeling and Passivity Analysis for Semi-Autonomous Multi-Robot Navigation in Three Dimensions","authors":"Takeshi Hatanaka;Takahiro Mochizuki;Takumi Sumino;José M. Maestre;Nikhil Chopra","doi":"10.1109/OJCSYS.2023.3343598","DOIUrl":"https://doi.org/10.1109/OJCSYS.2023.3343598","url":null,"abstract":"In this article, we study a one-human-multiple-robot interaction for human-enabled multi-robot navigation in three dimensions. We employ two fully distributed control architectures designed based on human passivity and human passivity shortage. The first half of this article focuses on human modeling and analysis for the passivity-based control architecture through human operation data on a 3-D human-in-the-loop simulator. Specifically, we compare virtual reality (VR) interfaces with a traditional interface, and examine the impacts that VR technology has on human properties in terms of model accuracy, performance, passivity and workload, demonstrating that VR interfaces have a positive effect on all aspects. In contrast to 1-D operation, we confirm that operators hardly attain passivity regardless of the network structure, even with the VR interfaces. We thus take the passivity-shortage-based control architecture and analyze the degree of passivity shortage. We then observe through user studies that operators tend to meet the degree of shortage needed to prove closed-loop stability.","PeriodicalId":73299,"journal":{"name":"IEEE open journal of control systems","volume":"3 ","pages":"45-57"},"PeriodicalIF":0.0,"publicationDate":"2023-12-15","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=10361530","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"139431037","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2023-12-06DOI: 10.1109/OJCSYS.2023.3340034
Sooyung Byeon;Joonwon Choi;Inseok Hwang
This article proposes a quantitative framework for optimally allocating task functions in human-autonomy teaming (HAT). HAT involves cooperation between humans and autonomous agents to achieve common goals. As humans and autonomous agents possess different capabilities, function allocation plays a crucial role in ensuring effective HAT. However, designing the best adaptive function allocation remains a challenge, as existing methods often rely on qualitative rules and intensive human-subject studies. To address this limitation, we propose a computational function allocation approach that leverages cognitive engineering, computational work models, and optimization techniques. The proposed optimal adaptive function allocation method is composed of three main elements: 1) analyze the teamwork to identify a set of all possible function allocations within a team construction, 2) numerically simulate the teamwork in temporal semantics to explore the interaction of the team with complex environments using the identified function allocations in a trial-and-error manner, and 3) optimize the adaptive function allocation with respect to a given situation such as physical conditions, available information resources, and human mental workload. For the optimization, we utilize performance metrics such as task performance, human mental workload, and coherency in function allocations. To illustrate the effectiveness of the proposed framework, we present a simulated HAT scenario involving a human work model and drone fleet for last-mile delivery in disaster relief operations.
本文提出了一个定量框架，用于优化人类-自主团队合作（HAT）中的任务功能分配。HAT 涉及人类与自主代理之间为实现共同目标而开展的合作。由于人类和自主代理拥有不同的能力，功能分配在确保有效的 HAT 中起着至关重要的作用。然而，设计最佳的自适应功能分配仍然是一项挑战，因为现有的方法通常依赖于定性规则和密集的人体研究。为了解决这一局限性，我们提出了一种计算功能分配方法，该方法利用了认知工程学、计算工作模型和优化技术。所提出的最优自适应功能分配方法由三个主要元素组成：1）分析团队工作，确定团队建设中所有可能的功能分配集合；2）在时间语义上对团队工作进行数值模拟，以试错方式探索团队与复杂环境的互动，并使用确定的功能分配；3）针对给定情况（如物理条件、可用信息资源和人的心理工作量）优化自适应功能分配。在优化过程中，我们利用任务性能、人的精神工作量和功能分配一致性等性能指标。为了说明所提框架的有效性，我们提出了一个模拟的 HAT 场景，其中涉及人类工作模型和无人机队，用于救灾行动中的最后一英里配送。
{"title":"A Computational Framework for Optimal Adaptive Function Allocation in a Human-Autonomy Teaming Scenario","authors":"Sooyung Byeon;Joonwon Choi;Inseok Hwang","doi":"10.1109/OJCSYS.2023.3340034","DOIUrl":"https://doi.org/10.1109/OJCSYS.2023.3340034","url":null,"abstract":"This article proposes a quantitative framework for optimally allocating task functions in human-autonomy teaming (HAT). HAT involves cooperation between humans and autonomous agents to achieve common goals. As humans and autonomous agents possess different capabilities, function allocation plays a crucial role in ensuring effective HAT. However, designing the best adaptive function allocation remains a challenge, as existing methods often rely on qualitative rules and intensive human-subject studies. To address this limitation, we propose a computational function allocation approach that leverages cognitive engineering, computational work models, and optimization techniques. The proposed optimal adaptive function allocation method is composed of three main elements: 1) analyze the teamwork to identify a set of all possible function allocations within a team construction, 2) numerically simulate the teamwork in temporal semantics to explore the interaction of the team with complex environments using the identified function allocations in a trial-and-error manner, and 3) optimize the adaptive function allocation with respect to a given situation such as physical conditions, available information resources, and human mental workload. For the optimization, we utilize performance metrics such as task performance, human mental workload, and coherency in function allocations. To illustrate the effectiveness of the proposed framework, we present a simulated HAT scenario involving a human work model and drone fleet for last-mile delivery in disaster relief operations.","PeriodicalId":73299,"journal":{"name":"IEEE open journal of control systems","volume":"3 ","pages":"32-44"},"PeriodicalIF":0.0,"publicationDate":"2023-12-06","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=10345767","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"139041247","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2023-11-08DOI: 10.1109/OJCSYS.2023.3331193
Hafiz Ahsan Said;Demián García-Violini;Nicolás Faedo;John V. Ringwood
Optimal control of wave energy converters (WECs), while converting wave energy into a usable form, such as electricity, may inject (reactive) power into the system at various points in the wave cycle. Though somewhat counter-intuitive, this action usually results in improved overall energy conversion. However, recent experimental results show that, on occasion, reactive power peaks can be significantly in excess of active power levels, leaving device developers with difficult decision in how to rate the power take-off of the system i.e. whether to cater for these high reactive power peaks, or limit power flow to rated (active) levels. The origins of these excessive power peaks are currently poorly understood, creating significant uncertainty in how to deal with them. In this paper, we show that, using both theoretical results and an illustrative simulation case study, under matched controller conditions (impedance-matching optimal condition), for both monochromatic and panchromatic sea-states, that the maximum peak reactive/active power ratio never exceeds unity . However, under mismatched WEC/controller conditions, this peak power ratio can exceed unity, bringing unrealistic demands on the power take-off (PTO) rating. The paper examines the various origins of system/controller mismatch, including modelling error, controller synthesis inaccuracies, and non-ideal PTO behaviour, highlighting the consequences of such errors on reactive power flow levels. This important result points to the need for accurate WEC modeling, while also showing the folly of catering for excessive reactive power peaks.
{"title":"On the Ratio of Reactive to Active Power in Wave Energy Converter Control","authors":"Hafiz Ahsan Said;Demián García-Violini;Nicolás Faedo;John V. Ringwood","doi":"10.1109/OJCSYS.2023.3331193","DOIUrl":"10.1109/OJCSYS.2023.3331193","url":null,"abstract":"Optimal control of wave energy converters (WECs), while converting wave energy into a usable form, such as electricity, may \u0000<italic>inject</i>\u0000 (reactive) power into the system at various points in the wave cycle. Though somewhat counter-intuitive, this action usually results in improved overall energy conversion. However, recent experimental results show that, on occasion, reactive power peaks can be significantly in excess of active power levels, leaving device developers with difficult decision in how to rate the power take-off of the system i.e. whether to cater for these high reactive power peaks, or limit power flow to rated (active) levels. The origins of these excessive power peaks are currently poorly understood, creating significant uncertainty in how to deal with them. In this paper, we show that, using both theoretical results and an illustrative simulation case study, \u0000<italic>under matched controller conditions</i>\u0000 (impedance-matching optimal condition), for both monochromatic and panchromatic sea-states, that the maximum peak reactive/active power ratio \u0000<italic>never exceeds unity</i>\u0000. However, under mismatched WEC/controller conditions, this peak power ratio can exceed unity, bringing unrealistic demands on the power take-off (PTO) rating. The paper examines the various origins of system/controller mismatch, including modelling error, controller synthesis inaccuracies, and non-ideal PTO behaviour, highlighting the consequences of such errors on reactive power flow levels. This important result points to the need for accurate WEC modeling, while also showing the folly of catering for excessive reactive power peaks.","PeriodicalId":73299,"journal":{"name":"IEEE open journal of control systems","volume":"3 ","pages":"14-31"},"PeriodicalIF":0.0,"publicationDate":"2023-11-08","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=10313027","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"135560786","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2023-11-08DOI: 10.1109/OJCSYS.2023.3331195
Yuzhen Qin;Alberto Maria Nobili;Danielle S. Bassett;Fabio Pasqualetti
Cluster synchronization is of great importance for the normal functioning of numerous technological and natural systems. Deviations from normal cluster synchronization patterns are closely associated with various malfunctions, such as neurological disorders in the brain. Therefore, it is crucial to restore normal system functions by stabilizing the appropriate cluster synchronization patterns. Most existing studies focus on designing controllers based on state measurements to achieve system stabilization. However, in many real-world scenarios, measuring system states in real time, such as neuronal activity in the brain, poses significant challenges, rendering the stabilization of such systems difficult. To overcome this challenge, in this article, we employ an open-loop control strategy, vibrational control , which does not require any state measurements. We establish some sufficient conditions under which vibrational inputs stabilize cluster synchronization. Further, we provide a tractable approach to design vibrational control. Finally, numerical experiments are conducted to demonstrate our theoretical findings.
{"title":"Vibrational Stabilization of Cluster Synchronization in Oscillator Networks","authors":"Yuzhen Qin;Alberto Maria Nobili;Danielle S. Bassett;Fabio Pasqualetti","doi":"10.1109/OJCSYS.2023.3331195","DOIUrl":"10.1109/OJCSYS.2023.3331195","url":null,"abstract":"Cluster synchronization is of great importance for the normal functioning of numerous technological and natural systems. Deviations from normal cluster synchronization patterns are closely associated with various malfunctions, such as neurological disorders in the brain. Therefore, it is crucial to restore normal system functions by stabilizing the appropriate cluster synchronization patterns. Most existing studies focus on designing controllers based on state measurements to achieve system stabilization. However, in many real-world scenarios, measuring system states in real time, such as neuronal activity in the brain, poses significant challenges, rendering the stabilization of such systems difficult. To overcome this challenge, in this article, we employ an open-loop control strategy, \u0000<italic>vibrational control</i>\u0000, which does not require any state measurements. We establish some sufficient conditions under which vibrational inputs stabilize cluster synchronization. Further, we provide a tractable approach to design vibrational control. Finally, numerical experiments are conducted to demonstrate our theoretical findings.","PeriodicalId":73299,"journal":{"name":"IEEE open journal of control systems","volume":"2 ","pages":"439-453"},"PeriodicalIF":0.0,"publicationDate":"2023-11-08","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=10313029","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"135559909","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2023-11-01DOI: 10.1109/OJCSYS.2023.3329394
Abolfazl Lavaei;Mateo Perez;Milad Kazemi;Fabio Somenzi;Sadegh Soudjani;Ashutosh Trivedi;Majid Zamani
We propose a compositional approach to synthesize policies for networks of continuous-space stochastic control systems with unknown dynamics using model-free reinforcement learning (RL). The approach is based on implicitly abstracting each subsystem in the network with a finite Markov decision process with unknown transition probabilities, synthesizing a strategy for each abstract model in an assume-guarantee fashion using RL, and then mapping the results back over the original network with approximate optimality guarantees. We provide lower bounds on the satisfaction probability of the overall network based on those over individual subsystems. A key contribution is to leverage the convergence results for adversarial RL (minimax Q-learning) on finite stochastic arenas to provide control strategies maximizing the probability of satisfaction over the network of continuous-space systems. We consider finite-horizon properties expressed in the syntactically co-safe fragment of linear temporal logic. These properties can readily be converted into automata-based reward functions, providing scalar reward signals suitable for RL. Since such reward functions are often sparse, we supply a potential-based reward shaping technique to accelerate learning by producing dense rewards. The effectiveness of the proposed approaches is demonstrated via two physical benchmarks including regulation of a room temperature network and control of a road traffic network.
{"title":"Compositional Reinforcement Learning for Discrete-Time Stochastic Control Systems","authors":"Abolfazl Lavaei;Mateo Perez;Milad Kazemi;Fabio Somenzi;Sadegh Soudjani;Ashutosh Trivedi;Majid Zamani","doi":"10.1109/OJCSYS.2023.3329394","DOIUrl":"10.1109/OJCSYS.2023.3329394","url":null,"abstract":"We propose a compositional approach to synthesize policies for networks of continuous-space stochastic control systems with unknown dynamics using model-free reinforcement learning (RL). The approach is based on \u0000<italic>implicitly</i>\u0000 abstracting each subsystem in the network with a finite Markov decision process with \u0000<italic>unknown</i>\u0000 transition probabilities, synthesizing a strategy for each abstract model in an assume-guarantee fashion using RL, and then mapping the results back over the original network with \u0000<italic>approximate optimality</i>\u0000 guarantees. We provide lower bounds on the satisfaction probability of the overall network based on those over individual subsystems. A key contribution is to leverage the convergence results for adversarial RL (minimax Q-learning) on finite stochastic arenas to provide control strategies maximizing the probability of satisfaction over the network of continuous-space systems. We consider \u0000<italic>finite-horizon</i>\u0000 properties expressed in the syntactically co-safe fragment of linear temporal logic. These properties can readily be converted into automata-based reward functions, providing scalar reward signals suitable for RL. Since such reward functions are often sparse, we supply a potential-based \u0000<italic>reward shaping</i>\u0000 technique to accelerate learning by producing dense rewards. The effectiveness of the proposed approaches is demonstrated via two physical benchmarks including regulation of a room temperature network and control of a road traffic network.","PeriodicalId":73299,"journal":{"name":"IEEE open journal of control systems","volume":"2 ","pages":"425-438"},"PeriodicalIF":0.0,"publicationDate":"2023-11-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=10304199","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"135362318","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
京公网安备 11010802042870号
京ICP备2023020795号-1
扫码关注我们
求助内容：
应助结果提醒方式：