Model-Free Tracking Control for Linear Stochastic Systems via Integral Reinforcement Learning

Abstract

This paper investigates optimal tracking control of linear stochastic systems with multiplicative state-dependent and input-dependent noise via a novel model-free integral reinforcement learning algorithm. We have conquered two major obstacles in this work. Firstly, the model-free tracking control on linear stochastic systems has scarcely been pivoted academically and it demands innovative methods and mathematical strategies to progress. An augmented stochastic differential equation of Itô’s type has been constructed while the control objective has been equated to minimization of the expected quadratic cost function. A model-dependent algorithm is shown to solve the stochastic algebraic Riccati equation. Second, control of linear stochastic systems essentially values for its sophisticated formation and sustained application. Inspired by optimal stationary control for linear stochastic systems, an integral reinforcement learning algorithm utilizing the adaptive dynamic programming method has been developed to scrap the reliance on the complete knowledge of system dynamics. The convergence of the core matrix and the sequence of control policies and the system stability has been rigorously studied following. Finally, numerical simulations are performed to demonstrate the efficacy of the proposed integral reinforcement learning methodology. Note to Practitioners—This paper focuses on the problem of optimal tracking control on linear stochastic systems, which is motivated by optimal stationary control and robust adaptive dynamic programming. Stochastic multiplicative noises are pervasive in modern control systems and engineering fields, such as power systems, aerospace systems, and industrial processes, thus important in modeling the random perturbation in system parameters and coefficients. However, research on tracking control on linear stochastic systems are challenging and languishing. First, we need to eradicate the dependence on complete knowledge system dynamics since machinery parameters in industrial design are constantly varying through time and space which builds a lasting impact on its structure, performance, and growth trajectory. Second, we innovatively apply integral reinforcement learning techniques on linear stochastic systems, which can be seamlessly integrated into networked control systems with noisy communication channels or neuronal networks to constructively solve stochastic multiplayer differential games online. Numerical simulations demonstrate that the proposed algorithm is feasible, which has also facilitated the compounding of robust adaptive dynamic programming and optimal tracking control on the industrial society. This study exhaustively presents the proposed approach to iteratively find optimal policies of the optimal tracking control problem directly from input and state data without explicitly identifying any system dynamics, and thoroughly unveils the explicit mathematical proof of convergence and system stability.