Single-Loop Deep Actor-Critic for Constrained Reinforcement Learning With Provable Convergence

Abstract

Deep actor-critic (DAC) algorithms, which combine actor-critic with deep neural network (DNN), have been among the most prevalent reinforcement learning algorithms for decision-making problems in simulated environments. However, the existing DAC algorithms are still not mature to solve realistic problems with non-convex stochastic constraints and high cost to interact with the environment. In this paper, we propose a single-loop DAC (SLDAC) algorithmic framework for general constrained reinforcement learning problems. In the actor module, the constrained stochastic successive convex approximation (CSSCA) method is applied to better handle the non-convex stochastic objective and constraints. In the critic module, the critic DNNs are only updated once or a few finite times for each iteration, which simplifies the algorithm to a single-loop framework. Moreover, the variance of the policy gradient estimation is reduced by reusing observations from the old policy. The single-loop design and the observation reuse effectively reduce the agent-environment interaction cost and computational complexity. Despite the biased policy gradient estimation incurred by the single-loop design and observation reuse, we prove that the SLDAC with a feasible initial point can converge to a Karush-Kuhn-Tuker (KKT) point of the original problem almost surely. Simulations show that the SLDAC algorithm can achieve superior performance with much lower interaction cost.