Inverse reinforcement learning by expert imitation for the stochastic linear–quadratic optimal control problem

Abstract

This article studies inverse reinforcement learning (IRL) for the linear–quadratic stochastic optimal control problem, where two agents are considered. A learner agent lacks knowledge of the expert agent’s cost function, but it reconstructs an underlying cost function by observing the expert agent’s states and controls, thereby imitating the expert agent’s optimal feedback control. We initially present a model-based IRL method, which consists of a policy correction and a policy update from the policy iteration in reinforcement learning, as well as a cost function weight reconstruction informed by the inverse optimal control. Afterward, under this scheme, we propose a model-free off-policy IRL method, which requires no system identification, only collecting behavior data from the learner agent and expert agent once during the iteration process. Moreover, the proofs of the method’s convergence, stability, and non-unique solutions are given. Finally, a numerical example and an inverse mean–variance portfolio optimization example are provided to validate the effectiveness of the presented method.