Pub Date : 2023-06-26DOI: 10.1109/OJCSYS.2023.3289771
Chen-Hao Chang;Jonathan Casas;Victor H. Duenas
Lower-limb exoskeletons can aid restoring mobility in people with movement disorders. Cable-driven exoskeletons can offload their actuators away from the human body to reduce the weight imposed on the user and enable precise control of joints. However, ensuring limb coordination through bidirectional motion control of joints using cables raise the technical challenge of preventing the occurrence of undesired cable slackness or counteracting forces between cables. Thus, motivation exists to develop a control design framework that integrates both a joint control loop to ensure suitable limb tracking and a cable control loop to maintain cable tension properly. In this article, a two-layer control structure consisting of high and low-level controllers are developed to ensure a knee-joint exoskeleton system follows the desired joint trajectories and adjusts the cable tension, respectively. A repetitive learning controller is designed for the high-level knee joint tracking objective motivated by the periodic nature of the desired leg swings (i.e., to achieve knee flexion and extension). Low-level robust controllers are developed for a pair of cables, each actuated by an electric motor, to track target motor trajectories composed of motor kinematics and offset angles to mitigate cable slackness. The offset angles are computed using admittance models that exploit measurements of the cable tensions as inputs. Each electric motor switches its role between tracking the knee joint trajectory (i.e., the motor acts as the leader motor to achieve flexion or extension) and implementing the low-level controller (i.e., the motor acts as the follower motor to reduce slackness). Hence, at any time, one motor is the leader and the other is the follower. A Lyapunov-based stability analysis is developed for the high-level joint controller to ensure global asymptotic tracking and the low-level follower controller to guarantee global exponential tracking. The designed controllers are implemented during leg swing experiments in six able-bodied individuals while wearing the knee joint cable-driven exoskeleton. A comparison of the results obtained in two trials with and without using the admittance model (i.e., exploiting cable tension measurements) is presented. The experimental results indicate improved knee joint tracking performance, smaller control input magnitudes, and reduced cable slackness in the trial that leveraged cable tension feedback compared to the trial that did not exploit tension feedback.
{"title":"Closed-Loop Kinematic and Indirect Force Control of a Cable-Driven Knee Exoskeleton: A Lyapunov-Based Switched Systems Approach","authors":"Chen-Hao Chang;Jonathan Casas;Victor H. Duenas","doi":"10.1109/OJCSYS.2023.3289771","DOIUrl":"https://doi.org/10.1109/OJCSYS.2023.3289771","url":null,"abstract":"Lower-limb exoskeletons can aid restoring mobility in people with movement disorders. Cable-driven exoskeletons can offload their actuators away from the human body to reduce the weight imposed on the user and enable precise control of joints. However, ensuring limb coordination through bidirectional motion control of joints using cables raise the technical challenge of preventing the occurrence of undesired cable slackness or counteracting forces between cables. Thus, motivation exists to develop a control design framework that integrates both a joint control loop to ensure suitable limb tracking and a cable control loop to maintain cable tension properly. In this article, a two-layer control structure consisting of high and low-level controllers are developed to ensure a knee-joint exoskeleton system follows the desired joint trajectories and adjusts the cable tension, respectively. A repetitive learning controller is designed for the high-level knee joint tracking objective motivated by the periodic nature of the desired leg swings (i.e., to achieve knee flexion and extension). Low-level robust controllers are developed for a pair of cables, each actuated by an electric motor, to track target motor trajectories composed of motor kinematics and offset angles to mitigate cable slackness. The offset angles are computed using admittance models that exploit measurements of the cable tensions as inputs. Each electric motor switches its role between tracking the knee joint trajectory (i.e., the motor acts as the leader motor to achieve flexion or extension) and implementing the low-level controller (i.e., the motor acts as the follower motor to reduce slackness). Hence, at any time, one motor is the leader and the other is the follower. A Lyapunov-based stability analysis is developed for the high-level joint controller to ensure global asymptotic tracking and the low-level follower controller to guarantee global exponential tracking. The designed controllers are implemented during leg swing experiments in six able-bodied individuals while wearing the knee joint cable-driven exoskeleton. A comparison of the results obtained in two trials with and without using the admittance model (i.e., exploiting cable tension measurements) is presented. The experimental results indicate improved knee joint tracking performance, smaller control input magnitudes, and reduced cable slackness in the trial that leveraged cable tension feedback compared to the trial that did not exploit tension feedback.","PeriodicalId":73299,"journal":{"name":"IEEE open journal of control systems","volume":"2 ","pages":"171-184"},"PeriodicalIF":0.0,"publicationDate":"2023-06-26","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://ieeexplore.ieee.org/iel7/9552933/9973428/10163824.pdf","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"50376115","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2023-06-02DOI: 10.1109/OJCSYS.2023.3282438
Per Sebastian Skardal;Alex Arenas
Robust coordination and organization in large ensembles of nonlinear oscillatory units play a vital role in a wide range of natural and engineered system. The control of self-organizing network-coupled systems has recently seen significant attention, but largely in the context of modifying or augmenting existing structures. This leaves a gap in our understanding of reactive control, where and how to design direct interventions, and what we may learn about structure and dynamics from such control strategies. Here we study reactive control of coupled oscillator networks and demonstrate dual control strategies, i.e., two different mechanisms for control, that may each be implemented on their own and interchangeably to achieve synchronization. These diverse strategies exploit different network properties, with the first directly targeting oscillators that are challenging to entrain, and the second focusing on oscillators with a strong influence on others. Thus, in addition to presenting alternative strategies for network control, the distinct control sets illuminate the oscillators' dynamical and structural roles within the system. The applicability of dual control is demonstrated using both synthetic and real networks.
{"title":"Dual Control of Coupled Oscillator Networks","authors":"Per Sebastian Skardal;Alex Arenas","doi":"10.1109/OJCSYS.2023.3282438","DOIUrl":"https://doi.org/10.1109/OJCSYS.2023.3282438","url":null,"abstract":"Robust coordination and organization in large ensembles of nonlinear oscillatory units play a vital role in a wide range of natural and engineered system. The control of self-organizing network-coupled systems has recently seen significant attention, but largely in the context of modifying or augmenting existing structures. This leaves a gap in our understanding of reactive control, where and how to design direct interventions, and what we may learn about structure and dynamics from such control strategies. Here we study reactive control of coupled oscillator networks and demonstrate dual control strategies, i.e., two different mechanisms for control, that may each be implemented on their own and interchangeably to achieve synchronization. These diverse strategies exploit different network properties, with the first directly targeting oscillators that are challenging to entrain, and the second focusing on oscillators with a strong influence on others. Thus, in addition to presenting alternative strategies for network control, the distinct control sets illuminate the oscillators' dynamical and structural roles within the system. The applicability of dual control is demonstrated using both synthetic and real networks.","PeriodicalId":73299,"journal":{"name":"IEEE open journal of control systems","volume":"2 ","pages":"146-154"},"PeriodicalIF":0.0,"publicationDate":"2023-06-02","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://ieeexplore.ieee.org/iel7/9552933/9973428/10143236.pdf","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"50376174","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2023-04-10DOI: 10.1109/OJCSYS.2023.3265901
Nicholas Rober;Sydney M. Katz;Chelsea Sidrane;Esen Yel;Michael Everett;Mykel J. Kochenderfer;Jonathan P. How
As neural networks (NNs) become more prevalent in safety-critical applications such as control of vehicles, there is a growing need to certify that systems with NN components are safe. This paper presents a set of backward reachability approaches for safety certification of neural feedback loops (NFLs), i.e., closed-loop systems with NN control policies. While backward reachability strategies have been developed for systems without NN components, the nonlinearities in NN activation functions and general noninvertibility of NN weight matrices make backward reachability for NFLs a challenging problem. To avoid the difficulties associated with propagating sets backward through NNs, we introduce a framework that leverages standard forward NN analysis tools to efficiently find over-approximations to backprojection (BP) sets, i.e., sets of states for which an NN policy will lead a system to a given target set. We present frameworks for calculating BP over-approximations for both linear and nonlinear systems with control policies represented by feedforward NNs and propose computationally efficient strategies. We use numerical results from a variety of models to showcase the proposed algorithms, including a demonstration of safety certification for a 6D system.
{"title":"Backward Reachability Analysis of Neural Feedback Loops: Techniques for Linear and Nonlinear Systems","authors":"Nicholas Rober;Sydney M. Katz;Chelsea Sidrane;Esen Yel;Michael Everett;Mykel J. Kochenderfer;Jonathan P. How","doi":"10.1109/OJCSYS.2023.3265901","DOIUrl":"https://doi.org/10.1109/OJCSYS.2023.3265901","url":null,"abstract":"As neural networks (NNs) become more prevalent in safety-critical applications such as control of vehicles, there is a growing need to certify that systems with NN components are safe. This paper presents a set of backward reachability approaches for safety certification of neural feedback loops (NFLs), i.e., closed-loop systems with NN control policies. While backward reachability strategies have been developed for systems without NN components, the nonlinearities in NN activation functions and general noninvertibility of NN weight matrices make backward reachability for NFLs a challenging problem. To avoid the difficulties associated with propagating sets backward through NNs, we introduce a framework that leverages standard forward NN analysis tools to efficiently find over-approximations to backprojection (BP) sets, i.e., sets of states for which an NN policy will lead a system to a given target set. We present frameworks for calculating BP over-approximations for both linear and nonlinear systems with control policies represented by feedforward NNs and propose computationally efficient strategies. We use numerical results from a variety of models to showcase the proposed algorithms, including a demonstration of safety certification for a 6D system.","PeriodicalId":73299,"journal":{"name":"IEEE open journal of control systems","volume":"2 ","pages":"108-124"},"PeriodicalIF":0.0,"publicationDate":"2023-04-10","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://ieeexplore.ieee.org/iel7/9552933/9973428/10097878.pdf","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"50376173","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2023-03-20DOI: 10.1109/OJCSYS.2023.3259228
Federico Celi;Giacomo Baggio;Fabio Pasqualetti
Imperfect models lead to imperfect controllers and deriving accurate models from first principles or system identification is especially challenging in networked systems. Instead, data can be used to directly compute controllers, without requiring any system identification or modeling. In this paper we propose a strategy to directly learn control actions when data from past system trajectories is distributed among multiple agents in a network. The approach we develop provably converges to a suboptimal solution in a finite number of steps, bounded by the diameter of the network, and with a sub-optimality gap that can be characterized as a function of data, and that can be made arbitrarily small. We further characterize the robustness properties of our approach and give provable guarantees on its performance when data are affected by noise or by a class of attacks.
{"title":"Distributed Data-Driven Control of Network Systems","authors":"Federico Celi;Giacomo Baggio;Fabio Pasqualetti","doi":"10.1109/OJCSYS.2023.3259228","DOIUrl":"https://doi.org/10.1109/OJCSYS.2023.3259228","url":null,"abstract":"Imperfect models lead to imperfect controllers and deriving accurate models from first principles or system identification is especially challenging in networked systems. Instead, data can be used to directly compute controllers, without requiring any system identification or modeling. In this paper we propose a strategy to directly learn control actions when data from past system trajectories is distributed among multiple agents in a network. The approach we develop provably converges to a suboptimal solution in a finite number of steps, bounded by the diameter of the network, and with a sub-optimality gap that can be characterized as a function of data, and that can be made arbitrarily small. We further characterize the robustness properties of our approach and give provable guarantees on its performance when data are affected by noise or by a class of attacks.","PeriodicalId":73299,"journal":{"name":"IEEE open journal of control systems","volume":"2 ","pages":"93-107"},"PeriodicalIF":0.0,"publicationDate":"2023-03-20","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://ieeexplore.ieee.org/iel7/9552933/9973428/10076813.pdf","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"50376172","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2023-03-18DOI: 10.1109/OJCSYS.2023.3277760
Mert Kayaalp;Fatima Ghadieh;Ali H. Sayed
Most works on multi-agent reinforcement learning focus on scenarios where the state of the environment is fully observable. In this work, we consider a cooperative policy evaluation task in which agents are not assumed to observe the environment state directly. Instead, agents can only have access to noisy observations and to belief vectors. It is well-known that finding global posterior distributions under multi-agent settings is generally NP-hard. As a remedy, we propose a fully decentralized belief forming strategy that relies on individual updates and on localized interactions over a communication network. In addition to the exchange of the beliefs, agents exploit the communication network by exchanging value function parameter estimates as well. We analytically show that the proposed strategy allows information to diffuse over the network, which in turn allows the agents' parameters to have a bounded difference with a centralized baseline. A multi-sensor target tracking application is considered in the simulations.
{"title":"Policy Evaluation in Decentralized POMDPs With Belief Sharing","authors":"Mert Kayaalp;Fatima Ghadieh;Ali H. Sayed","doi":"10.1109/OJCSYS.2023.3277760","DOIUrl":"https://doi.org/10.1109/OJCSYS.2023.3277760","url":null,"abstract":"Most works on multi-agent reinforcement learning focus on scenarios where the state of the environment is fully observable. In this work, we consider a cooperative policy evaluation task in which agents are not assumed to observe the environment state directly. Instead, agents can only have access to noisy observations and to belief vectors. It is well-known that finding global posterior distributions under multi-agent settings is generally NP-hard. As a remedy, we propose a fully decentralized belief forming strategy that relies on individual updates and on localized interactions over a communication network. In addition to the exchange of the beliefs, agents exploit the communication network by exchanging value function parameter estimates as well. We analytically show that the proposed strategy allows information to diffuse over the network, which in turn allows the agents' parameters to have a bounded difference with a centralized baseline. A multi-sensor target tracking application is considered in the simulations.","PeriodicalId":73299,"journal":{"name":"IEEE open journal of control systems","volume":"2 ","pages":"125-145"},"PeriodicalIF":0.0,"publicationDate":"2023-03-18","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://ieeexplore.ieee.org/iel7/9552933/9973428/10129007.pdf","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"50226357","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2023-03-17DOI: 10.1109/OJCSYS.2023.3277308
Hany Abdulsamad;Jan Peters
Optimal control of general nonlinear systems is a central challenge in automation. Enabled by powerful function approximators, data-driven approaches to control have recently successfully tackled challenging applications. However, such methods often obscure the structure of dynamics and control behind black-box over-parameterized representations, thus limiting our ability to understand closed-loop behavior. This article adopts a hybrid-system view of nonlinear modeling and control that lends an explicit hierarchical structure to the problem and breaks down complex dynamics into simpler localized units. We consider a sequence modeling paradigm that captures the temporal structure of the data and derive an expectation-maximization (EM) algorithm that automatically decomposes nonlinear dynamics into stochastic piecewise affine models with nonlinear transition boundaries. Furthermore, we show that these time-series models naturally admit a closed-loop extension that we use to extract local polynomial feedback controllers from nonlinear experts via behavioral cloning. Finally, we introduce a novel hybrid relative entropy policy search (Hb-REPS) technique that incorporates the hierarchical nature of hybrid models and optimizes a set of time-invariant piecewise feedback controllers derived from a piecewise polynomial approximation of a global state-value function.
{"title":"Model-Based Reinforcement Learning via Stochastic Hybrid Models","authors":"Hany Abdulsamad;Jan Peters","doi":"10.1109/OJCSYS.2023.3277308","DOIUrl":"https://doi.org/10.1109/OJCSYS.2023.3277308","url":null,"abstract":"Optimal control of general nonlinear systems is a central challenge in automation. Enabled by powerful function approximators, data-driven approaches to control have recently successfully tackled challenging applications. However, such methods often obscure the structure of dynamics and control behind black-box over-parameterized representations, thus limiting our ability to understand closed-loop behavior. This article adopts a hybrid-system view of nonlinear modeling and control that lends an explicit hierarchical structure to the problem and breaks down complex dynamics into simpler localized units. We consider a sequence modeling paradigm that captures the temporal structure of the data and derive an expectation-maximization (EM) algorithm that automatically decomposes nonlinear dynamics into stochastic piecewise affine models with nonlinear transition boundaries. Furthermore, we show that these time-series models naturally admit a closed-loop extension that we use to extract local polynomial feedback controllers from nonlinear experts via behavioral cloning. Finally, we introduce a novel hybrid relative entropy policy search (Hb-REPS) technique that incorporates the hierarchical nature of hybrid models and optimizes a set of time-invariant piecewise feedback controllers derived from a piecewise polynomial approximation of a global state-value function.","PeriodicalId":73299,"journal":{"name":"IEEE open journal of control systems","volume":"2 ","pages":"155-170"},"PeriodicalIF":0.0,"publicationDate":"2023-03-17","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://ieeexplore.ieee.org/iel7/9552933/9973428/10128705.pdf","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"50376175","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
While reinforcement learning produces very promising results for many applications, its main disadvantage is the lack of safety guarantees, which prevents its use in safety-critical systems. In this work, we address this issue by a safety shield for nonlinear continuous systems that solve reach-avoid tasks. Our safety shield prevents applying potentially unsafe actions from a reinforcement learning agent by projecting the proposed action to the closest safe action. This approach is called action projection and is implemented via mixed-integer optimization. The safety constraints for action projection are obtained by applying parameterized reachability analysis using polynomial zonotopes, which enables to accurately capture the nonlinear effects of the actions on the system. In contrast to other state-of-the-art approaches for action projection, our safety shield can efficiently handle input constraints and dynamic obstacles, eases incorporation of the spatial robot dimensions into the safety constraints, guarantees robust safety despite process noise and measurement errors, and is well suited for high-dimensional systems, as we demonstrate on several challenging benchmark systems.
{"title":"Provably Safe Reinforcement Learning via Action Projection Using Reachability Analysis and Polynomial Zonotopes","authors":"Niklas Kochdumper;Hanna Krasowski;Xiao Wang;Stanley Bak;Matthias Althoff","doi":"10.1109/OJCSYS.2023.3256305","DOIUrl":"https://doi.org/10.1109/OJCSYS.2023.3256305","url":null,"abstract":"While reinforcement learning produces very promising results for many applications, its main disadvantage is the lack of safety guarantees, which prevents its use in safety-critical systems. In this work, we address this issue by a safety shield for nonlinear continuous systems that solve reach-avoid tasks. Our safety shield prevents applying potentially unsafe actions from a reinforcement learning agent by projecting the proposed action to the closest safe action. This approach is called action projection and is implemented via mixed-integer optimization. The safety constraints for action projection are obtained by applying parameterized reachability analysis using polynomial zonotopes, which enables to accurately capture the nonlinear effects of the actions on the system. In contrast to other state-of-the-art approaches for action projection, our safety shield can efficiently handle input constraints and dynamic obstacles, eases incorporation of the spatial robot dimensions into the safety constraints, guarantees robust safety despite process noise and measurement errors, and is well suited for high-dimensional systems, as we demonstrate on several challenging benchmark systems.","PeriodicalId":73299,"journal":{"name":"IEEE open journal of control systems","volume":"2 ","pages":"79-92"},"PeriodicalIF":0.0,"publicationDate":"2023-03-13","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://ieeexplore.ieee.org/iel7/9552933/9973428/10068193.pdf","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"50376171","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2023-02-27DOI: 10.1109/OJCSYS.2023.3250089
Babak Salamat;Gerhard Elsbacher;Andrea M. Tonello;Lenz Belzner
One of the challenging problems in sensor network systems is to estimate and track the state of a target point mass with unknown dynamics. Recent improvements in deep learning (DL) show a renewed interest in applying DL techniques to state estimation problems. However, the process noise is absent which seems to indicate that the point-mass target must be non-maneuvering, as process noise is typically as significant as the measurement noise for tracking maneuvering targets. In this paper, we propose a continuous-time (CT) model-free or model-building distributed reinforcement learning estimator (DRLE) using an integral value function in sensor networks. The DRLE algorithm is capable of learning an optimal policy from a neural value function that aims to provide the estimation of a target point mass. The proposed estimator consists of two high pass consensus filters in terms of weighted measurements and inverse-covariance matrices and a critic reinforcement learning mechanism for each node in the network. The efficiency of the proposed DRLE is shown by a simulation experiment of a network of underactuated vertical takeoff and landing aircraft with strong input coupling. The experiment highlights two advantages of DRLE: i) it does not require the dynamic model to be known, and ii) it is an order of magnitude faster than the state-dependent Riccati equation (SDRE) baseline.
{"title":"Model-Free Distributed Reinforcement Learning State Estimation of a Dynamical System Using Integral Value Functions","authors":"Babak Salamat;Gerhard Elsbacher;Andrea M. Tonello;Lenz Belzner","doi":"10.1109/OJCSYS.2023.3250089","DOIUrl":"https://doi.org/10.1109/OJCSYS.2023.3250089","url":null,"abstract":"One of the challenging problems in sensor network systems is to estimate and track the state of a target point mass with unknown dynamics. Recent improvements in deep learning (DL) show a renewed interest in applying DL techniques to state estimation problems. However, the process noise is absent which seems to indicate that the point-mass target must be non-maneuvering, as process noise is typically as significant as the measurement noise for tracking maneuvering targets. In this paper, we propose a continuous-time (CT) model-free or model-building distributed reinforcement learning estimator (DRLE) using an integral value function in sensor networks. The DRLE algorithm is capable of learning an optimal policy from a neural value function that aims to provide the estimation of a target point mass. The proposed estimator consists of two high pass consensus filters in terms of weighted measurements and inverse-covariance matrices and a critic reinforcement learning mechanism for each node in the network. The efficiency of the proposed DRLE is shown by a simulation experiment of a network of underactuated vertical takeoff and landing aircraft with strong input coupling. The experiment highlights two advantages of DRLE: i) it does not require the dynamic model to be known, and ii) it is an order of magnitude faster than the state-dependent Riccati equation (SDRE) baseline.","PeriodicalId":73299,"journal":{"name":"IEEE open journal of control systems","volume":"2 ","pages":"70-78"},"PeriodicalIF":0.0,"publicationDate":"2023-02-27","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://ieeexplore.ieee.org/iel7/9552933/9973428/10054475.pdf","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"50376170","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2023-02-10DOI: 10.1109/OJCSYS.2023.3244089
Ankit Gupta;Mustafa Khammash
The well-known Internal Model Principle (IMP) is a cornerstone of modern control theory. It stipulates the necessary conditions for asymptotic robustness of disturbance-prone dynamical systems by asserting that such a system must embed a subsystem in a feedback loop, and this subsystem must be able to reduplicate the dynamic disturbance using only the regulated variable as the input. The insights provided by IMP can help in both designing suitable controllers and also in analysing the regulatory mechanisms in complex systems. So far the application of IMP in biology has been case-specific and ad hoc, primarily due to the lack of generic versions of the IMP for biomolecular reaction networks that model biological processes. In this short article we highlight the need for an IMP in biology and discuss a recently developed version of it for biomolecular networks that exhibit maximal Robust Perfect Adaptation (maxRPA) by being robust to the maximum number of disturbance sources.
{"title":"The Internal Model Principle for Biomolecular Control Theory","authors":"Ankit Gupta;Mustafa Khammash","doi":"10.1109/OJCSYS.2023.3244089","DOIUrl":"https://doi.org/10.1109/OJCSYS.2023.3244089","url":null,"abstract":"The well-known Internal Model Principle (IMP) is a cornerstone of modern control theory. It stipulates the necessary conditions for asymptotic robustness of disturbance-prone dynamical systems by asserting that such a system must embed a subsystem in a feedback loop, and this subsystem must be able to reduplicate the dynamic disturbance using only the regulated variable as the input. The insights provided by IMP can help in both designing suitable controllers and also in analysing the regulatory mechanisms in complex systems. So far the application of IMP in biology has been case-specific and ad hoc, primarily due to the lack of generic versions of the IMP for biomolecular reaction networks that model biological processes. In this short article we highlight the need for an IMP in biology and discuss a recently developed version of it for biomolecular networks that exhibit maximal Robust Perfect Adaptation (maxRPA) by being robust to the maximum number of disturbance sources.","PeriodicalId":73299,"journal":{"name":"IEEE open journal of control systems","volume":"2 ","pages":"63-69"},"PeriodicalIF":0.0,"publicationDate":"2023-02-10","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://ieeexplore.ieee.org/iel7/9552933/9973428/10041993.pdf","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"50226358","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2023-02-01DOI: 10.1109/OJCSYS.2023.3241486
Tongxin Li;Ruixiao Yang;Guannan Qu;Yiheng Lin;Adam Wierman;Steven H. Low
Machine-learned black-box policies are ubiquitous for nonlinear control problems. Meanwhile, crude model information is often available for these problems from, e.g., linear approximations of nonlinear dynamics. We study the problem of certifying a black-box control policy with stability using model-based advice for nonlinear control on a single trajectory. We first show a general negative result that a naive convex combination of a black-box policy and a linear model-based policy can lead to instability, even if the two policies are both stabilizing. We then propose an adaptive $lambda$-confident policy, with a coefficient $lambda$ indicating the confidence in a black-box policy, and prove its stability. With bounded nonlinearity, in addition, we show that the adaptive $lambda$-confident policy achieves a bounded competitive ratio when a black-box policy is near-optimal. Finally, we propose an online learning approach to implement the adaptive $lambda$-confident policy and verify its efficacy in case studies about the Cart-Pole problem and a real-world electric vehicle (EV) charging problem with covariate shift due to COVID-19.
{"title":"Certifying Black-Box Policies With Stability for Nonlinear Control","authors":"Tongxin Li;Ruixiao Yang;Guannan Qu;Yiheng Lin;Adam Wierman;Steven H. Low","doi":"10.1109/OJCSYS.2023.3241486","DOIUrl":"https://doi.org/10.1109/OJCSYS.2023.3241486","url":null,"abstract":"Machine-learned black-box policies are ubiquitous for nonlinear control problems. Meanwhile, crude model information is often available for these problems from, e.g., linear approximations of nonlinear dynamics. We study the problem of certifying a black-box control policy with stability using model-based advice for nonlinear control on a single trajectory. We first show a general negative result that a naive convex combination of a black-box policy and a linear model-based policy can lead to instability, even if the two policies are both stabilizing. We then propose an \u0000<italic>adaptive <inline-formula><tex-math>$lambda$</tex-math></inline-formula>-confident policy</i>\u0000, with a coefficient \u0000<inline-formula><tex-math>$lambda$</tex-math></inline-formula>\u0000 indicating the confidence in a black-box policy, and prove its stability. With bounded nonlinearity, in addition, we show that the adaptive \u0000<inline-formula><tex-math>$lambda$</tex-math></inline-formula>\u0000-confident policy achieves a bounded competitive ratio when a black-box policy is near-optimal. Finally, we propose an online learning approach to implement the adaptive \u0000<inline-formula><tex-math>$lambda$</tex-math></inline-formula>\u0000-confident policy and verify its efficacy in case studies about the Cart-Pole problem and a real-world electric vehicle (EV) charging problem with covariate shift due to COVID-19.","PeriodicalId":73299,"journal":{"name":"IEEE open journal of control systems","volume":"2 ","pages":"49-62"},"PeriodicalIF":0.0,"publicationDate":"2023-02-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://ieeexplore.ieee.org/iel7/9552933/9973428/10034859.pdf","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"50376169","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
京公网安备 11010802042870号
京ICP备2023020795号-1
扫码关注我们
求助内容：
应助结果提醒方式：