Pub Date : 2024-04-15DOI: 10.1109/OJCSYS.2024.3388374
Alessio Maritan;Luca Schenato;Subhrakanti Dey
The Hessian matrix conveys important information about the curvature, spectrum and partial derivatives of a function, and is required in a variety of tasks. However, computing the exact Hessian is prohibitively expensive for high-dimensional input spaces, and is just impossible in zeroth-order optimization, where the objective function is a black-box of which only input-output pairs are known. In this work we address this relevant problem by providing a rigorous analysis of an Hessian estimator available in the literature, allowing it to be used as a provably accurate replacement of the true Hessian matrix. The Hessian estimator is randomized and incremental, and its computation requires only point function evaluations. We provide non-asymptotic convergence bounds on the estimation error and derive the minimum number of function queries needed to achieve a desired accuracy with arbitrarily high probability. In the second part of the paper we show a practical application of our results, introducing a novel optimization algorithm suitable for non-convex and black-box federated learning. The algorithm only requires clients to evaluate their local functions at certain input points, and builds a sufficiently accurate estimate of the global Hessian matrix in a distributed way. The algorithm exploits inexact cubic regularization to escape saddle points and guarantees convergence with optimal iteration complexity and high probability. Numerical results show that the proposed algorithm outperforms the existing zeroth-order federated algorithms in both convex and non-convex problems. Furthermore, we achieve similar performance to state-of-the-art algorithms for federated convex optimization that use exact gradients and Hessian matrices.
{"title":"Novel Bounds for Incremental Hessian Estimation With Application to Zeroth-Order Federated Learning","authors":"Alessio Maritan;Luca Schenato;Subhrakanti Dey","doi":"10.1109/OJCSYS.2024.3388374","DOIUrl":"https://doi.org/10.1109/OJCSYS.2024.3388374","url":null,"abstract":"The Hessian matrix conveys important information about the curvature, spectrum and partial derivatives of a function, and is required in a variety of tasks. However, computing the exact Hessian is prohibitively expensive for high-dimensional input spaces, and is just impossible in zeroth-order optimization, where the objective function is a black-box of which only input-output pairs are known. In this work we address this relevant problem by providing a rigorous analysis of an Hessian estimator available in the literature, allowing it to be used as a provably accurate replacement of the true Hessian matrix. The Hessian estimator is randomized and incremental, and its computation requires only point function evaluations. We provide non-asymptotic convergence bounds on the estimation error and derive the minimum number of function queries needed to achieve a desired accuracy with arbitrarily high probability. In the second part of the paper we show a practical application of our results, introducing a novel optimization algorithm suitable for non-convex and black-box federated learning. The algorithm only requires clients to evaluate their local functions at certain input points, and builds a sufficiently accurate estimate of the global Hessian matrix in a distributed way. The algorithm exploits inexact cubic regularization to escape saddle points and guarantees convergence with optimal iteration complexity and high probability. Numerical results show that the proposed algorithm outperforms the existing zeroth-order federated algorithms in both convex and non-convex problems. Furthermore, we achieve similar performance to state-of-the-art algorithms for federated convex optimization that use exact gradients and Hessian matrices.","PeriodicalId":73299,"journal":{"name":"IEEE open journal of control systems","volume":"3 ","pages":"173-189"},"PeriodicalIF":0.0,"publicationDate":"2024-04-15","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=10499850","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"140818786","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-04-04DOI: 10.1109/OJCSYS.2024.3385348
Lars Lindemann;Alexander Robey;Lejun Jiang;Satyajeet Das;Stephen Tu;Nikolai Matni
This paper addresses learning safe output feedback control laws from partial observations of expert demonstrations. We assume that a model of the system dynamics and a state estimator are available along with corresponding error bounds, e.g., estimated from data in practice. We first propose robust output control barrier functions (ROCBFs) as a means to guarantee safety, as defined through controlled forward invariance of a safe set. We then formulate an optimization problem to learn ROCBFs from expert demonstrations that exhibit safe system behavior, e.g., data collected from a human operator or an expert controller. When the parametrization of the ROCBF is linear, then we show that, under mild assumptions, the optimization problem is convex. Along with the optimization problem, we provide verifiable conditions in terms of the density of the data, smoothness of the system model and state estimator, and the size of the error bounds that guarantee validity of the obtained ROCBF. Towards obtaining a practical control algorithm, we propose an algorithmic implementation of our theoretical framework that accounts for assumptions made in our framework in practice. We validate our algorithm in the autonomous driving simulator CARLA and demonstrate how to learn safe control laws from simulated RGB camera images.
{"title":"Learning Robust Output Control Barrier Functions From Safe Expert Demonstrations","authors":"Lars Lindemann;Alexander Robey;Lejun Jiang;Satyajeet Das;Stephen Tu;Nikolai Matni","doi":"10.1109/OJCSYS.2024.3385348","DOIUrl":"https://doi.org/10.1109/OJCSYS.2024.3385348","url":null,"abstract":"This paper addresses learning safe output feedback control laws from partial observations of expert demonstrations. We assume that a model of the system dynamics and a state estimator are available along with corresponding error bounds, e.g., estimated from data in practice. We first propose robust output control barrier functions (ROCBFs) as a means to guarantee safety, as defined through controlled forward invariance of a safe set. We then formulate an optimization problem to learn ROCBFs from expert demonstrations that exhibit safe system behavior, e.g., data collected from a human operator or an expert controller. When the parametrization of the ROCBF is linear, then we show that, under mild assumptions, the optimization problem is convex. Along with the optimization problem, we provide verifiable conditions in terms of the density of the data, smoothness of the system model and state estimator, and the size of the error bounds that guarantee validity of the obtained ROCBF. Towards obtaining a practical control algorithm, we propose an algorithmic implementation of our theoretical framework that accounts for assumptions made in our framework in practice. We validate our algorithm in the autonomous driving simulator CARLA and demonstrate how to learn safe control laws from simulated RGB camera images.","PeriodicalId":73299,"journal":{"name":"IEEE open journal of control systems","volume":"3 ","pages":"158-172"},"PeriodicalIF":0.0,"publicationDate":"2024-04-04","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=10491341","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"140818787","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-03-31DOI: 10.1109/OJCSYS.2024.3407999
Abhishek Pandala;Aaron D. Ames;Kaveh Akbari Hamed
This paper formally develops robust optimal predictive control solutions that can accommodate disturbances and stabilize periodic legged locomotion. To this end, we build upon existing optimization-based control paradigms, particularly quadratic programming (QP)-based model predictive controllers (MPCs). We present conditions under which the closed-loop reduced-order systems (i.e., template models) with MPC have the continuous differentiability property on an open neighborhood of gaits. We then linearize the resulting discrete-time, closed-loop nonlinear template system around the gait to obtain a linear time-varying (LTV) system. This periodic LTV system is further transformed into a linear system with a constant state-transition matrix using discrete-time Floquet transform. The system is then analyzed to accommodate parametric uncertainties and to synthesize robust optimal $mathcal {H}_{2}$ and $mathcal {H}_infty$ feedback controllers via linear matrix inequalities (LMIs). The paper then extends the theoretical results to the single rigid body (SRB) template dynamics and numerically verifies them. The proposed robust optimal predictive controllers are used in a layered control structure, where the optimal reduced-order trajectories are provided to a full-order nonlinear whole-body controller (WBC) for tracking at the low level. The developed layered controllers are numerically and experimentally validated for the robust locomotion of the A1 quadrupedal robot subject to various disturbances and uneven terrains. Our numerical results suggest that the $mathcal {H}_{2}$ - and $mathcal {H}_infty$ -optimal MPC controllers significantly improve the robust stability of the gaits compared to the normal MPC.
{"title":"$mathcal {H}_{2}$- and $mathcal {H}_infty$-Optimal Model Predictive Controllers for Robust Legged Locomotion","authors":"Abhishek Pandala;Aaron D. Ames;Kaveh Akbari Hamed","doi":"10.1109/OJCSYS.2024.3407999","DOIUrl":"https://doi.org/10.1109/OJCSYS.2024.3407999","url":null,"abstract":"This paper formally develops robust optimal predictive control solutions that can accommodate disturbances and stabilize periodic legged locomotion. To this end, we build upon existing optimization-based control paradigms, particularly quadratic programming (QP)-based model predictive controllers (MPCs). We present conditions under which the closed-loop reduced-order systems (i.e., template models) with MPC have the continuous differentiability property on an open neighborhood of gaits. We then linearize the resulting discrete-time, closed-loop nonlinear template system around the gait to obtain a linear time-varying (LTV) system. This periodic LTV system is further transformed into a linear system with a constant state-transition matrix using discrete-time Floquet transform. The system is then analyzed to accommodate parametric uncertainties and to synthesize robust optimal \u0000<inline-formula><tex-math>$mathcal {H}_{2}$</tex-math></inline-formula>\u0000 and \u0000<inline-formula><tex-math>$mathcal {H}_infty$</tex-math></inline-formula>\u0000 feedback controllers via linear matrix inequalities (LMIs). The paper then extends the theoretical results to the single rigid body (SRB) template dynamics and numerically verifies them. The proposed robust optimal predictive controllers are used in a layered control structure, where the optimal reduced-order trajectories are provided to a full-order nonlinear whole-body controller (WBC) for tracking at the low level. The developed layered controllers are numerically and experimentally validated for the robust locomotion of the A1 quadrupedal robot subject to various disturbances and uneven terrains. Our numerical results suggest that the \u0000<inline-formula><tex-math>$mathcal {H}_{2}$</tex-math></inline-formula>\u0000- and \u0000<inline-formula><tex-math>$mathcal {H}_infty$</tex-math></inline-formula>\u0000-optimal MPC controllers significantly improve the robust stability of the gaits compared to the normal MPC.","PeriodicalId":73299,"journal":{"name":"IEEE open journal of control systems","volume":"3 ","pages":"225-238"},"PeriodicalIF":0.0,"publicationDate":"2024-03-31","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=10543084","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141315179","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
An innovative solution to the optimal motion planning problem is presented in this work. A novel parametrized actor structure is proposed, which guarantees safe and convergent navigation by construction. Concurrently, an efficient scheme for optimizing a mixed state and energy cost function is formulated. The proposed method inherits the positive traits of continuous methods, while at the same time providing sub-optimal –but close to optimal– results significantly faster and in more complex workspaces than previous ones. The scheme is demonstrated to outperform established relevant methods, while at the same time being competitive w.r.t. execution time. Extensive simulations to validate the effectiveness of the method are presented, along with relevant technical proofs for safety and convergence.
{"title":"An Efficient Solution to Optimal Motion Planning With Provable Safety and Convergence","authors":"PANAGIOTIS ROUSSEAS;Charalampos Bechlioulis;Kostas Kyriakopoulos","doi":"10.1109/OJCSYS.2024.3378055","DOIUrl":"https://doi.org/10.1109/OJCSYS.2024.3378055","url":null,"abstract":"An innovative solution to the optimal motion planning problem is presented in this work. A novel parametrized actor structure is proposed, which guarantees safe and convergent navigation by construction. Concurrently, an efficient scheme for optimizing a mixed state and energy cost function is formulated. The proposed method inherits the positive traits of continuous methods, while at the same time providing sub-optimal –but close to optimal– results significantly faster and in more complex workspaces than previous ones. The scheme is demonstrated to outperform established relevant methods, while at the same time being competitive w.r.t. execution time. Extensive simulations to validate the effectiveness of the method are presented, along with relevant technical proofs for safety and convergence.","PeriodicalId":73299,"journal":{"name":"IEEE open journal of control systems","volume":"3 ","pages":"143-157"},"PeriodicalIF":0.0,"publicationDate":"2024-03-18","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=10473133","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"140345509","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-03-16DOI: 10.1109/OJCSYS.2024.3402050
Juan A. Paredes;Omran Kouba;Dennis S. Bernstein
Self-excited systems (SES) arise in numerous applications, such as fluid-structure interaction, combustion, and biochemical systems. In support of system identification and digital control of SES, this paper analyzes discrete-time Lur'e systems with affinely constrained, piecewise-C $^{1}$ feedback nonlinearities. In particular, a novel feature of the discrete-time Lur'e system considered in this paper is the structural assumption that the linear dynamics possess a zero at 1. This assumption ensures that the Lur'e system have a unique equilibrium for each constant, exogenous input and prevents the system from having an additional equilibrium with a nontrivial domain of attraction. The main result provides sufficient conditions under which a discrete-time Lur'e system is self-excited in the sense that its response is 1) nonconvergent for almost all initial conditions, and 2) bounded for all initial conditions. Sufficient conditions for 1) include the instability and nonsingularity of the linearized, closed-loop dynamics at the unique equilibrium and their nonsingularity almost everywhere. Sufficient conditions for 2) include asymptotic stability of the linear dynamics of the Lur'e system and their feedback interconnection with linear mappings that correspond to the affine constraints that bound the nonlinearity, as well as the feasibility of a linear matrix inequality.
{"title":"Self-Excited Dynamics of Discrete-Time Lur'e Systems With Affinely Constrained, Piecewise-C$^{1}$ Feedback Nonlinearities","authors":"Juan A. Paredes;Omran Kouba;Dennis S. Bernstein","doi":"10.1109/OJCSYS.2024.3402050","DOIUrl":"https://doi.org/10.1109/OJCSYS.2024.3402050","url":null,"abstract":"Self-excited systems (SES) arise in numerous applications, such as fluid-structure interaction, combustion, and biochemical systems. In support of system identification and digital control of SES, this paper analyzes discrete-time Lur'e systems with affinely constrained, piecewise-C\u0000<inline-formula><tex-math>$^{1}$</tex-math></inline-formula>\u0000 feedback nonlinearities. In particular, a novel feature of the discrete-time Lur'e system considered in this paper is the structural assumption that the linear dynamics possess a zero at 1. This assumption ensures that the Lur'e system have a unique equilibrium for each constant, exogenous input and prevents the system from having an additional equilibrium with a nontrivial domain of attraction. The main result provides sufficient conditions under which a discrete-time Lur'e system is self-excited in the sense that its response is 1) nonconvergent for almost all initial conditions, and 2) bounded for all initial conditions. Sufficient conditions for 1) include the instability and nonsingularity of the linearized, closed-loop dynamics at the unique equilibrium and their nonsingularity almost everywhere. Sufficient conditions for 2) include asymptotic stability of the linear dynamics of the Lur'e system and their feedback interconnection with linear mappings that correspond to the affine constraints that bound the nonlinearity, as well as the feasibility of a linear matrix inequality.","PeriodicalId":73299,"journal":{"name":"IEEE open journal of control systems","volume":"3 ","pages":"214-224"},"PeriodicalIF":0.0,"publicationDate":"2024-03-16","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=10531633","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141304041","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-03-08DOI: 10.1109/OJCSYS.2024.3398530
Hao Chen;Abhishek Gupta;Yin Sun;Ness Shroff
This paper considers the change point detection problem under dependent samples. In particular, we provide performance guarantees for the MMD-CUSUM test under exponentially $alpha$ , $beta$ , and fast $phi$ -mixing processes, which significantly expands its utility beyond the i.i.d. and Markovian cases used in previous studies. We obtain lower bounds for average-run-length ( $ {mathtt {ARL}}$ ) and upper bounds for average-detection-delay ( $ {mathtt {ADD}}$ ) in terms of the threshold parameter. We show that the MMD-CUSUM test enjoys the same level of performance as the i.i.d. case under fast $phi$ -mixing processes. The MMD-CUSUM test also achieves strong performance under exponentially $alpha$ / $beta$ -mixing processes, which are significantly more relaxed than existing results. The MMD-CUSUM test statistic adapts to different settings without modifications, rendering it a completely data-driven, dependence-agnostic change point detection scheme. Numerical simulations are provided at the end to evaluate our findings.
{"title":"Model-Free Change Point Detection for Mixing Processes","authors":"Hao Chen;Abhishek Gupta;Yin Sun;Ness Shroff","doi":"10.1109/OJCSYS.2024.3398530","DOIUrl":"https://doi.org/10.1109/OJCSYS.2024.3398530","url":null,"abstract":"This paper considers the change point detection problem under dependent samples. In particular, we provide performance guarantees for the MMD-CUSUM test under exponentially \u0000<inline-formula><tex-math>$alpha$</tex-math></inline-formula>\u0000, \u0000<inline-formula><tex-math>$beta$</tex-math></inline-formula>\u0000, and fast \u0000<inline-formula><tex-math>$phi$</tex-math></inline-formula>\u0000-mixing processes, which significantly expands its utility beyond the i.i.d. and Markovian cases used in previous studies. We obtain lower bounds for average-run-length (\u0000<inline-formula><tex-math>$ {mathtt {ARL}}$</tex-math></inline-formula>\u0000) and upper bounds for average-detection-delay (\u0000<inline-formula><tex-math>$ {mathtt {ADD}}$</tex-math></inline-formula>\u0000) in terms of the threshold parameter. We show that the MMD-CUSUM test enjoys the same level of performance as the i.i.d. case under fast \u0000<inline-formula><tex-math>$phi$</tex-math></inline-formula>\u0000-mixing processes. The MMD-CUSUM test also achieves strong performance under exponentially \u0000<inline-formula><tex-math>$alpha$</tex-math></inline-formula>\u0000/\u0000<inline-formula><tex-math>$beta$</tex-math></inline-formula>\u0000-mixing processes, which are significantly more relaxed than existing results. The MMD-CUSUM test statistic adapts to different settings without modifications, rendering it a completely data-driven, dependence-agnostic change point detection scheme. Numerical simulations are provided at the end to evaluate our findings.","PeriodicalId":73299,"journal":{"name":"IEEE open journal of control systems","volume":"3 ","pages":"202-213"},"PeriodicalIF":0.0,"publicationDate":"2024-03-08","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=10522896","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141304042","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-03-06DOI: 10.1109/OJCSYS.2024.3397270
Gianluca Bianchin;Fabio Pasqualetti
Advanced traffic navigation systems, which provide routing recommendations to drivers based on real-time congestion information, are nowadays widely adopted by roadway transportation users. Yet, the emerging effects on the traffic dynamics originating from the widespread adoption of these technologies have remained largely unexplored until now. In this paper, we propose a dynamic model where drivers imitate the path preferences of previous drivers, and we study the properties of its equilibrium points. Our model is a dynamic generalization of the classical traffic assignment framework , and extends it by accounting for dynamics both in the path decision process and in the network's traffic flows. We show that, when travelers learn shortest paths by imitating other travelers, the overall traffic system benefits from this mechanism and transfers the maximum admissible amount of traffic demand. On the other hand, we demonstrate that, when the travel delay functions are not sufficiently steep or the rates at which drivers imitate previous travelers are not adequately chosen, the trajectories of the traffic system may fail to converge to an equilibrium point, thus compromising asymptotic stability. Illustrative numerical simulations combined with empirical data from highway sensors illustrate our findings.
{"title":"Navigation Systems May Deteriorate Stability in Traffic Networks","authors":"Gianluca Bianchin;Fabio Pasqualetti","doi":"10.1109/OJCSYS.2024.3397270","DOIUrl":"10.1109/OJCSYS.2024.3397270","url":null,"abstract":"Advanced traffic navigation systems, which provide routing recommendations to drivers based on real-time congestion information, are nowadays widely adopted by roadway transportation users. Yet, the emerging effects on the traffic dynamics originating from the widespread adoption of these technologies have remained largely unexplored until now. In this paper, we propose a dynamic model where drivers imitate the path preferences of previous drivers, and we study the properties of its equilibrium points. Our model is a dynamic generalization of the classical \u0000<italic>traffic assignment framework</i>\u0000, and extends it by accounting for dynamics both in the path decision process and in the network's traffic flows. We show that, when travelers learn shortest paths by imitating other travelers, the overall traffic system benefits from this mechanism and transfers the maximum admissible amount of traffic demand. On the other hand, we demonstrate that, when the travel delay functions are not sufficiently steep or the rates at which drivers imitate previous travelers are not adequately chosen, the trajectories of the traffic system may fail to converge to an equilibrium point, thus compromising asymptotic stability. Illustrative numerical simulations combined with empirical data from highway sensors illustrate our findings.","PeriodicalId":73299,"journal":{"name":"IEEE open journal of control systems","volume":"3 ","pages":"239-252"},"PeriodicalIF":0.0,"publicationDate":"2024-03-06","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=10520878","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141221343","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-03-05DOI: 10.1109/OJCSYS.2024.3372929
Lai Wei;Vaibhav Srivastava
We study the nonstationary stochastic Multi-Armed Bandit (MAB) problem in which the distributions of rewards associated with arms are assumed to be time-varying and the total variation in the expected rewards is subject to a variation budget. The regret of a policy is defined by the difference in the expected cumulative reward obtained using the policy and using an oracle that selects the arm with the maximum mean reward at each time. We characterize the performance of the proposed policies in terms of the worst-case regret, which is the supremum of the regret over the set of reward distribution sequences satisfying the variation budget. We design Upper-Confidence Bound (UCB)-based policies with three different approaches, namely, periodic resetting, sliding observation window, and discount factor, and show that they are order-optimal with respect to the minimax regret, i.e., the minimum worst-case regret achieved by any policy. We also relax the sub-Gaussian assumption on reward distributions and develop robust versions of the proposed policies that can handle heavy-tailed reward distributions and maintain their performance guarantees.
{"title":"Nonstationary Stochastic Bandits: UCB Policies and Minimax Regret","authors":"Lai Wei;Vaibhav Srivastava","doi":"10.1109/OJCSYS.2024.3372929","DOIUrl":"https://doi.org/10.1109/OJCSYS.2024.3372929","url":null,"abstract":"We study the nonstationary stochastic Multi-Armed Bandit (MAB) problem in which the distributions of rewards associated with arms are assumed to be time-varying and the total variation in the expected rewards is subject to a variation budget. The regret of a policy is defined by the difference in the expected cumulative reward obtained using the policy and using an oracle that selects the arm with the maximum mean reward at each time. We characterize the performance of the proposed policies in terms of the worst-case regret, which is the supremum of the regret over the set of reward distribution sequences satisfying the variation budget. We design Upper-Confidence Bound (UCB)-based policies with three different approaches, namely, periodic resetting, sliding observation window, and discount factor, and show that they are order-optimal with respect to the minimax regret, i.e., the minimum worst-case regret achieved by any policy. We also relax the sub-Gaussian assumption on reward distributions and develop robust versions of the proposed policies that can handle heavy-tailed reward distributions and maintain their performance guarantees.","PeriodicalId":73299,"journal":{"name":"IEEE open journal of control systems","volume":"3 ","pages":"128-142"},"PeriodicalIF":0.0,"publicationDate":"2024-03-05","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=10460198","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"140342701","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-02-22DOI: 10.1109/OJCSYS.2024.3368850
Marjan Khaledi;Bahare Kiumarsi
This article presents a proactive approach to resolving the conflict between safety and optimality for continuous-time (CT) safety-critical systems with unknown dynamics. The presented method guarantees safety and performance specifications by combining two controllers: a safe controller and an optimal controller. On the one hand, the safe controller is designed using only input and state data measurements and without requiring the state derivative data, which are typically required in data-driven control of CT systems. State derivative measurement is costly, and its approximation introduces noise to the system. On the other hand, the optimal controller is learned using a low-complexity one-shot optimization problem, which again does not rely on prior knowledge of the system dynamics and state derivative data. Compared to existing optimal control learning methods for CT systems, which are typically iterative, a one-shot optimization is considerably more sample-efficient and computationally efficient. The share of optimal and safe controllers in the overall control policy is obtained by solving a computationally efficient optimization problem involving a scalar variable in a data-driven manner. It is shown that the contribution of the safe controller dominates that of the optimal controller when the system's state is close to the safety boundaries, and this domination drops as the system trajectories move away from the safety boundaries. In this case, the optimal controller contributes more to the overall controller. The feasibility and stability of the proposed controller are shown. Finally, the simulation results show the efficacy of the proposed approach.
{"title":"A Computationally-Efficient Data-Driven Safe Optimal Algorithm Through Control Merging","authors":"Marjan Khaledi;Bahare Kiumarsi","doi":"10.1109/OJCSYS.2024.3368850","DOIUrl":"https://doi.org/10.1109/OJCSYS.2024.3368850","url":null,"abstract":"This article presents a proactive approach to resolving the conflict between safety and optimality for continuous-time (CT) safety-critical systems with unknown dynamics. The presented method guarantees safety and performance specifications by combining two controllers: a safe controller and an optimal controller. On the one hand, the safe controller is designed using only input and state data measurements and without requiring the state derivative data, which are typically required in data-driven control of CT systems. State derivative measurement is costly, and its approximation introduces noise to the system. On the other hand, the optimal controller is learned using a low-complexity one-shot optimization problem, which again does not rely on prior knowledge of the system dynamics and state derivative data. Compared to existing optimal control learning methods for CT systems, which are typically iterative, a one-shot optimization is considerably more sample-efficient and computationally efficient. The share of optimal and safe controllers in the overall control policy is obtained by solving a computationally efficient optimization problem involving a scalar variable in a data-driven manner. It is shown that the contribution of the safe controller dominates that of the optimal controller when the system's state is close to the safety boundaries, and this domination drops as the system trajectories move away from the safety boundaries. In this case, the optimal controller contributes more to the overall controller. The feasibility and stability of the proposed controller are shown. Finally, the simulation results show the efficacy of the proposed approach.","PeriodicalId":73299,"journal":{"name":"IEEE open journal of control systems","volume":"3 ","pages":"118-127"},"PeriodicalIF":0.0,"publicationDate":"2024-02-22","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=10443513","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"140328981","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-01-26DOI: 10.1109/OJCSYS.2024.3359089
Shihao Xie;Giovanni Russo
We consider both leaderless and leader-follower, possibly nonlinear, networks affected by time-varying communication delays. For such systems, we give a set of sufficient conditions that guarantee the convergence of the network towards some desired behaviour while simultaneously ensuring the rejection of polynomial disturbances and the non-amplification of other classes of disturbances across the network. To fulfill these desired properties, and prove our main results, we propose the use of a control protocol that implements a multiplex architecture. The use of our results for control protocol design is then illustrated in the context of formation control. The protocols are validated both in-silico and via an experimental set-up with real robots. All experiments confirm the effectiveness of our approach.
{"title":"A Multiplex Approach Against Disturbance Propagation in Nonlinear Networks With Delays","authors":"Shihao Xie;Giovanni Russo","doi":"10.1109/OJCSYS.2024.3359089","DOIUrl":"https://doi.org/10.1109/OJCSYS.2024.3359089","url":null,"abstract":"We consider both leaderless and leader-follower, possibly nonlinear, networks affected by time-varying communication delays. For such systems, we give a set of sufficient conditions that guarantee the convergence of the network towards some desired behaviour while simultaneously ensuring the rejection of polynomial disturbances and the non-amplification of other classes of disturbances across the network. To fulfill these desired properties, and prove our main results, we propose the use of a control protocol that implements a multiplex architecture. The use of our results for control protocol design is then illustrated in the context of formation control. The protocols are validated both in-silico and via an experimental set-up with real robots. All experiments confirm the effectiveness of our approach.","PeriodicalId":73299,"journal":{"name":"IEEE open journal of control systems","volume":"3 ","pages":"87-101"},"PeriodicalIF":0.0,"publicationDate":"2024-01-26","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=10415106","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"139937154","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
京公网安备 11010802042870号
京ICP备2023020795号-1
扫码关注我们
求助内容：
应助结果提醒方式：