GAILPG: Multiagent Policy Gradient With Generative Adversarial Imitation Learning

Abstract

In reinforcement learning, the agents need to sufficiently explore the environment and efficiently exploit the existing experiences before finding the solution to the tasks, particularly in cooperative multiagent scenarios where the state and action spaces grow exponentially with the number of agents. Hence, enhancing the exploration ability of agents and improving the utilization efficiency of experiences are two critical issues in cooperative multiagent reinforcement learning. We propose a novel method called generative adversarial imitation learning policy gradients (GAILPG). The contributions of GAILPG are as follows: first, we integrate generative adversarial self-imitation learning into the multiagent actor–critic framework to improve the utilization efficiency of experiences, thus further assisting the policy training; second, we design a new curiosity module to enhance the exploration ability of the agents. Experimental results on the StarCraft II micromanagement benchmark demonstrate that GAILPG surpasses state-of-the-art policy-based methods and is even on par with the value-based methods and the ablation experiments validate the reasonability of the discriminator module and the curiosity module encapsulated in our method.