Model-based policy optimization algorithms for feedback control of complex dynamic systems

Abstract

Model-free Reinforcement learning (RL) has been successfully used in benchmark systems such as the Cart-Pole, Inverted-Pendulum, and Robotic arms. However, model-free RL algorithms have several limitations, including large data requirements and handling of state constraints. Model-based and hybrid RL algorithms offer opportunities to tackle these limitations. This research investigated the application of a model-based policy optimization algorithm (MBPO) for feedback control of the Van de Vusse reaction and the Quadruple tank system. MBPO-trained agents suffer from inaccuracies of the learned model and the computational burden of the online optimization neural network models and policy parameters. We propose a modified model-based policy optimization (MMBPO) algorithm that uses linear dynamic system models. This minimizes a learned model’s inaccuracies and eliminates the computational requirements of training the neural network models. Simulation results show that model-based policy optimization algorithms can track the setpoints of the dynamic systems studied.