Adaptive optimal controllers based on Generalized Policy Iteration in a continuous-time framework

In this paper we present two adaptive algorithms which offer solution to the continuous-time optimal control problem for nonlinear, affine in the inputs, time-invariant systems. Both algorithms were developed based on the Generalized Policy Iteration technique and involve adaptation of two neural network structures namely Actor, providing the control signal, and Critic, performing evaluation of the control performance. Despite the similarities, the two adaptive algorithms differ in the manner in which the adaptation takes place, required knowledge on the system dynamics, and formulation of the persistence of excitation requirement. The main difference is that one algorithm uses sequential adaptation of the actor and critic structures, i.e. while one is trained the other one is kept constant, while for the second algorithm the two neural networks are trained synchronously in a continuous-time fashion. The two algorithms are described in detail and proof of convergence is provided. Simulation results of applying the two algorithms for finding the optimal state feedback controller of a nonlinear system are also presented.