Advanced value iteration for discrete-time intelligent critic control: A survey

Optimal control problems are ubiquitous in practical engineering applications and social life with the idea of cost or resource conservation. Based on the critic learning scheme, adaptive dynamic programming (ADP) is regarded as a significant avenue to address the optimal control problems by combining the advanced design ideas such as adaptive control, reinforcement learning, and intelligent control. This survey introduces the recent development of ADP and related intelligent critic control with an emphasis on advanced value iteration (VI) schemes for discrete-time nonlinear systems. The theoretical results focus on convergence and stability properties for general VI, stabilizing VI, integrated VI, evolving VI, adjustable VI schemes and so on. Several significant applications are also elaborated in aspects of optimal regulation, optimal tracking, and zero-sum games. We aim to break through the bottleneck problems for VI algorithms in realizing evolving control, accelerating learning speed, and reducing the calculation expense. In addition, the prospects of new theoretical and technical fields for advanced VI schemes are looked ahead.