Policy distillation for efficient decentralized execution in multi-agent reinforcement learning

Abstract

Cooperative Multi-Agent Reinforcement Learning (MARL) addresses complex scenarios where multiple agents collaborate to achieve shared objectives. Training these agents in partially observable environments within the Centralized Training with Decentralized Execution (CTDE) framework remains challenging due to limited information access and the need for lightweight agent networks. To overcome these challenges, we introduce the Centralized Training and Policy Distillation for Decentralized Execution (CTPDE) framework. We propose a centralized dual-attention agent network that integrates global state and local observations to enable lossless value decomposition and prevent homogeneous agent behaviors. In addition, an efficient policy distillation method is proposed, in which a network of action value distribution is distilled from the centralized agent network, ensuring the efficiency of decentralized execution. The evaluation of CTPDE in benchmark environments demonstrates that the attention-based network achieves state-of-the-art performance during training. Moreover, the distilled agent network surpasses existing RNN-based methods and, in some cases, matches the capabilities of more complex architectures. These findings underscore the potential of CTPDE for advancing cooperative MARL tasks.