Interpretable multi-agent reinforcement learning via multi-head variational autoencoders

Abstract

Multi-agent deep reinforcement learning (RL) is increasingly proficient at making collective decisions in complex systems. However, the black-box nature of DRL decision networks often renders agent behaviors difficult to interpret, thereby undermining human trust. Although several reinforcement learning explanation methods have been proposed, most mainly identify factors influencing decisions without elucidating the underlying causal mechanisms based on physical models. Moreover, these methods do not address the generalizability of interpretability within multi-agent system settings. To overcome these challenges, we propose a multi-agent RL network based on multi-head variational autoencoders (MVAE), which generates decisions with interpretable physical semantics for unmanned systems. The MVAE directly encodes multiple types of semantically meaningful features with physical interpretations from the latent space and generates decisions by integrating these semantics according to physical models. Furthermore, considering the different latent variable distributions in continuous and discrete action scenarios, we design two distinct MVAE models based on Gaussian and Dirichlet distributions, respectively, and design training frameworks using deterministic policy gradient networks and proximal policy optimization networks in a multi-agent environment. Additionally, we develop a visualization method to intuitively convey interpretability in both continuous and discrete action scenarios. Simulation experiments comparing our method with existing baselines demonstrate that our approach achieves superior decision-making performance under interpretability conditions, and further validate its performance in large-scale scenarios.