Online Q-learning for stochastic linear systems with state and control dependent noise

Abstract

For continuous-time (CT) systems that are characterized by stochastic differential equations (SDEs) with completely unknown dynamics parameters, a reinforcement learning (RL)-based optimal control framework is presented in this paper. To obtain the near-optimal control policy, an online Q-learning algorithm is proposed by learning the data sampled from the system state trajectory, while an integral reinforcement learning (IRL) approach is developed in stochastic situation so as to formulate the Q-learning iterative algorithm. In particular, an actor/critic neural network (NN) structure is applied in iteration, where a critic approximator is used for estimating the designed Q-function while an actor approximator is for estimating the optimal control policy online. To ensure the convergence of iteration, the tuning laws of two neural networks are designed, respectively, by a gradient descent scheme. Moreover, the mean-square stability of the closed-loop system is proved through rigorous analysis, and the convergence to optimal solution is guaranteed as well.