Distributed Actor-Critic Learning Using Emphatic Weightings

Abstract

In this paper a new Actor-Critic algorithm is proposed for distributed off-policy multi-agent reinforcement learning. It is composed of the Emphatic Temporal Difference ETD${\left(\lambda \right)}$ algorithm (at the Critic stage) and a complementary distributed consensus-based algorithm using the exact gradients of a given criterion function (at the Actor stage). It is demonstrated that the algorithm converges weakly to the invariant set of an ordinary differential equation (ODE) characterizing the whole algorithm. Simulation results are presented as an illustration of high efficiency of the proposed algorithm.