Output Feedback H∞ Control of Unknown Discrete-time Linear Systems: Off-policy Reinforcement Learning

Abstract

In this paper, a data-driven output feedback approach is developed for solving H∞ control problem of linear discrete-time systems based on off-policy reinforcement learning (RL) algorithm. Past input-output measurements are leveraged to implicitly reconstruct the system's states to alleviate the requirement to measure or estimate the system's states. Then, an off-policy input-output Bellman equation is derived based on this implicit reconstruction to evaluate control policies using only input-output measurements. An improved control policy is then learned utilizing the solution to the Bellman equation without knowing the system's dynamics. In the proposed approach, unlike the on-policy methods, the disturbance does not need to be updated in a predefined manner at each iteration, which makes it more practical. While the state-feedback off-policy RL method is shown to be a bias-free approach for deterministic systems, it is shown that once the system's states have been reconstructed from the input-output measurements, the input-output off-policy method cannot be considered as an immune approach against the probing noises. To cope with this, a discount factor is utilized in the performance function to decay the deleterious effect of probing noises. Finally, to illustrate the sensitivity of the problem to the probing noises and the efficacy of the proposed approach, the flight control system is tested in the simulation.