{"title":"Reinforcement Learning for Finite-Horizon H∞ Tracking Control of Unknown Discrete Linear Time-Varying System","authors":"Linwei Ye;Zhonggai Zhao;Fei Liu","doi":"10.1109/TSMC.2024.3431453","DOIUrl":null,"url":null,"abstract":"This article considers the finite-horizon H\n<inline-formula> <tex-math>$_{\\infty }$ </tex-math></inline-formula>\n tracking problem for a class of discrete linear time-varying systems. Two reinforcement learning (RL) methods—policy iteration (PI) and Q-learning—are proposed to solve this problem. The latter can obtain the H\n<inline-formula> <tex-math>$_{\\infty }$ </tex-math></inline-formula>\n controller without system dynamics. In the field of RL control, most studies focus on infinite-horizon control and time-invariant systems, and few studies have investigated finite-horizon control or time-varying systems. In contrast to infinite-horizon H\n<inline-formula> <tex-math>$_{\\infty }$ </tex-math></inline-formula>\n tracking control, finite-horizon H\n<inline-formula> <tex-math>$_{\\infty }$ </tex-math></inline-formula>\n tracking control involves a time-varying value function. While this introduces challenges, it empowers the algorithm to effectively handle time-varying problems. Within the finite-horizon framework, the value function is bounded, allowing the removal of the discount factor, thereby enhancing control performance. Additionally, there is no longer a need for an admissible control law for initialization, providing the proposed algorithms with the combined advantages of both PI and value iteration (VI). Two simulation examples are used to verify the effectiveness of the proposed algorithms.","PeriodicalId":48915,"journal":{"name":"IEEE Transactions on Systems Man Cybernetics-Systems","volume":null,"pages":null},"PeriodicalIF":8.6000,"publicationDate":"2024-08-05","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"IEEE Transactions on Systems Man Cybernetics-Systems","FirstCategoryId":"94","ListUrlMain":"https://ieeexplore.ieee.org/document/10623340/","RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"AUTOMATION & CONTROL SYSTEMS","Score":null,"Total":0}
引用次数: 0
Abstract
This article considers the finite-horizon H
$_{\infty }$
tracking problem for a class of discrete linear time-varying systems. Two reinforcement learning (RL) methods—policy iteration (PI) and Q-learning—are proposed to solve this problem. The latter can obtain the H
$_{\infty }$
controller without system dynamics. In the field of RL control, most studies focus on infinite-horizon control and time-invariant systems, and few studies have investigated finite-horizon control or time-varying systems. In contrast to infinite-horizon H
$_{\infty }$
tracking control, finite-horizon H
$_{\infty }$
tracking control involves a time-varying value function. While this introduces challenges, it empowers the algorithm to effectively handle time-varying problems. Within the finite-horizon framework, the value function is bounded, allowing the removal of the discount factor, thereby enhancing control performance. Additionally, there is no longer a need for an admissible control law for initialization, providing the proposed algorithms with the combined advantages of both PI and value iteration (VI). Two simulation examples are used to verify the effectiveness of the proposed algorithms.
本文研究了一类离散线性时变系统的有限视距 H $_{\infty }$ 跟踪问题。提出了两种强化学习(RL)方法--策略迭代(PI)和 Q 学习--来解决这个问题。后者可以在不考虑系统动态的情况下获得 H $_{\infty }$ 控制器。在 RL 控制领域,大多数研究集中于无限视距控制和时变系统,很少有研究涉及有限视距控制或时变系统。与无限视距 H $_{\infty }$ 跟踪控制不同,有限视距 H $_{\infty }$ 跟踪控制涉及时变值函数。虽然这带来了挑战,但却使算法能够有效处理时变问题。在有限视距框架内,价值函数是有界的,因此可以去除贴现因子,从而提高控制性能。此外,初始化时不再需要可接受的控制律,从而使所提出的算法兼具 PI 和值迭代 (VI) 的优点。两个仿真实例用于验证所提算法的有效性。
期刊介绍:
The IEEE Transactions on Systems, Man, and Cybernetics: Systems encompasses the fields of systems engineering, covering issue formulation, analysis, and modeling throughout the systems engineering lifecycle phases. It addresses decision-making, issue interpretation, systems management, processes, and various methods such as optimization, modeling, and simulation in the development and deployment of large systems.