Pub Date : 2020-03-06DOI: 10.3929/ETHZ-B-000404690
Napat Karnchanachari, M. I. Valls, David Hoeller, M. Hutter
Model Predictive Control (MPC) is a powerful control technique that handles constraints, takes the system's dynamics into account, and optimizes for a given cost function. In practice, however, it often requires an expert to craft and tune this cost function and find trade-offs between different state penalties to satisfy simple high level objectives. In this paper, we use Reinforcement Learning and in particular value learning to approximate the value function given only high level objectives, which can be sparse and binary. Building upon previous works, we present improvements that allowed us to successfully deploy the method on a real world unmanned ground vehicle. Our experiments show that our method can learn the cost function from scratch and without human intervention, while reaching a performance level similar to that of an expert-tuned MPC. We perform a quantitative comparison of these methods with standard MPC approaches both in simulation and on the real robot.
{"title":"Practical Reinforcement Learning For MPC: Learning from sparse objectives in under an hour on a real robot","authors":"Napat Karnchanachari, M. I. Valls, David Hoeller, M. Hutter","doi":"10.3929/ETHZ-B-000404690","DOIUrl":"https://doi.org/10.3929/ETHZ-B-000404690","url":null,"abstract":"Model Predictive Control (MPC) is a powerful control technique that handles constraints, takes the system's dynamics into account, and optimizes for a given cost function. In practice, however, it often requires an expert to craft and tune this cost function and find trade-offs between different state penalties to satisfy simple high level objectives. In this paper, we use Reinforcement Learning and in particular value learning to approximate the value function given only high level objectives, which can be sparse and binary. Building upon previous works, we present improvements that allowed us to successfully deploy the method on a real world unmanned ground vehicle. Our experiments show that our method can learn the cost function from scratch and without human intervention, while reaching a performance level similar to that of an expert-tuned MPC. We perform a quantitative comparison of these methods with standard MPC approaches both in simulation and on the real robot.","PeriodicalId":268449,"journal":{"name":"Conference on Learning for Dynamics & Control","volume":"58 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2020-03-06","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"115548097","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Policy optimization (PO) is a key ingredient for reinforcement learning (RL). For control design, certain constraints are usually enforced on the policies to optimize, accounting for either the stability, robustness, or safety concerns on the system. Hence, PO is by nature a constrained (nonconvex) optimization in most cases, whose global convergence is challenging to analyze in general. More importantly, some constraints that are safety-critical, e.g., the $mathcal{H}_infty$-norm constraint that guarantees the system robustness, are difficult to enforce as the PO methods proceed. Recently, policy gradient methods have been shown to converge to the global optimum of linear quadratic regulator (LQR), a classical optimal control problem, without regularizing/projecting the control iterates onto the stabilizing set (Fazel et al., 2018), its (implicit) feasible set. This striking result is built upon the coercive property of the cost, ensuring that the iterates remain feasible as the cost decreases. In this paper, we study the convergence theory of PO for $mathcal{H}_2$ linear control with $mathcal{H}_infty$-norm robustness guarantee. One significant new feature of this problem is the lack of coercivity, i.e., the cost may have finite value around the feasible set boundary, breaking the existing analysis for LQR. Interestingly, we show that two PO methods enjoy the implicit regularization property, i.e., the iterates preserve the $mathcal{H}_infty$ robustness constraint as if they are regularized by the algorithms. Furthermore, convergence to the globally optimal policies with globally sublinear and locally (super-)linear rates are provided under certain conditions, despite the nonconvexity of the problem. To the best of our knowledge, our work offers the first results on the implicit regularization property and global convergence of PO methods for robust/risk-sensitive control.
策略优化(PO)是强化学习(RL)的关键组成部分。对于控制设计，通常在策略上施加某些约束以进行优化，以考虑系统的稳定性、健壮性或安全性问题。因此，在大多数情况下，PO本质上是一种约束(非凸)优化，其全局收敛性通常很难分析。更重要的是，一些对安全至关重要的约束，例如，保证系统健壮性的$mathcal{H}_infty$ -norm约束，在PO方法进行时很难强制执行。最近，策略梯度方法已被证明收敛到线性二次型调节器(LQR)的全局最优，这是一个经典的最优控制问题，而不需要将控制迭代正则化/投影到稳定集(Fazel et al.， 2018)，即它的(隐式)可行集上。这个惊人的结果建立在成本的强制属性上，确保迭代在成本降低时仍然可行。本文研究了具有$mathcal{H}_infty$ -范数鲁棒性保证的$mathcal{H}_2$线性控制的PO收敛理论。该问题的一个重要新特征是缺乏矫顽力，即成本在可行集边界附近可能具有有限值，打破了现有的LQR分析。有趣的是，我们证明了两个PO方法具有隐式正则化特性，即迭代保留$mathcal{H}_infty$鲁棒性约束，就像它们被算法正则化一样。此外，尽管问题具有非凸性，但在一定条件下，给出了具有全局次线性和局部超线性速率的全局最优策略的收敛性。据我们所知，我们的工作提供了关于鲁棒/风险敏感控制的PO方法的隐式正则化性质和全局收敛性的第一个结果。
{"title":"Policy Optimization for H2 Linear Control with H∞ Robustness Guarantee: Implicit Regularization and Global Convergence","authors":"K. Zhang, Bin Hu, T. Başar","doi":"10.1137/20m1347942","DOIUrl":"https://doi.org/10.1137/20m1347942","url":null,"abstract":"Policy optimization (PO) is a key ingredient for reinforcement learning (RL). For control design, certain constraints are usually enforced on the policies to optimize, accounting for either the stability, robustness, or safety concerns on the system. Hence, PO is by nature a constrained (nonconvex) optimization in most cases, whose global convergence is challenging to analyze in general. More importantly, some constraints that are safety-critical, e.g., the $mathcal{H}_infty$-norm constraint that guarantees the system robustness, are difficult to enforce as the PO methods proceed. Recently, policy gradient methods have been shown to converge to the global optimum of linear quadratic regulator (LQR), a classical optimal control problem, without regularizing/projecting the control iterates onto the stabilizing set (Fazel et al., 2018), its (implicit) feasible set. This striking result is built upon the coercive property of the cost, ensuring that the iterates remain feasible as the cost decreases. In this paper, we study the convergence theory of PO for $mathcal{H}_2$ linear control with $mathcal{H}_infty$-norm robustness guarantee. One significant new feature of this problem is the lack of coercivity, i.e., the cost may have finite value around the feasible set boundary, breaking the existing analysis for LQR. Interestingly, we show that two PO methods enjoy the implicit regularization property, i.e., the iterates preserve the $mathcal{H}_infty$ robustness constraint as if they are regularized by the algorithms. Furthermore, convergence to the globally optimal policies with globally sublinear and locally (super-)linear rates are provided under certain conditions, despite the nonconvexity of the problem. To the best of our knowledge, our work offers the first results on the implicit regularization property and global convergence of PO methods for robust/risk-sensitive control.","PeriodicalId":268449,"journal":{"name":"Conference on Learning for Dynamics & Control","volume":"136 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2019-10-21","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"133650190","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 1900-01-01DOI: 10.48550/arXiv.2211.15923
Alex Devonport, P. Seiler, M. Arcak
Complex-valued Gaussian processes are used in Bayesian frequency-domain system identification as prior models for regression. If each realization of such a process were an H∞ function with probability one, then the same model could be used for probabilistic robust control, allowing for robustly safe learning. We investigate sufficient conditions for a general complex-domain Gaussian process to have this property. For the special case of processes whose Hermitian covariance is stationary, we provide an explicit parameterization of the covariance structure in terms of a summable sequence of nonnegative numbers.
{"title":"Frequency Domain Gaussian Process Models for H∞ Uncertainties","authors":"Alex Devonport, P. Seiler, M. Arcak","doi":"10.48550/arXiv.2211.15923","DOIUrl":"https://doi.org/10.48550/arXiv.2211.15923","url":null,"abstract":"Complex-valued Gaussian processes are used in Bayesian frequency-domain system identification as prior models for regression. If each realization of such a process were an H∞ function with probability one, then the same model could be used for probabilistic robust control, allowing for robustly safe learning. We investigate sufficient conditions for a general complex-domain Gaussian process to have this property. For the special case of processes whose Hermitian covariance is stationary, we provide an explicit parameterization of the covariance structure in terms of a summable sequence of nonnegative numbers.","PeriodicalId":268449,"journal":{"name":"Conference on Learning for Dynamics & Control","volume":"21 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"1900-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"126924624","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
京公网安备 11010802042870号
京ICP备2023020795号-1
扫码关注我们
求助内容：
应助结果提醒方式：