A Hetero-Relation Transformer Network for Multiagent Reinforcement Learning

Abstract

Recently, considerable research has been focused on multiagent reinforcement learning to effectively account for each agent's relations. However, most research has focused on homogeneous multiagent systems with the same type of agents, which has limited application to heterogeneous multiagent systems. The demand for heterogeneous systems has considerably increased not only in games but also in the real world. Therefore, a technique that can properly consider relations in heterogeneous systems is required. In this article, we propose a novel transformer network called HRformer, which is based on heterogeneous graph networks that can reflect the heterogeneity and relations among agents. To this end, we design an effective linear encoding method for the transformer to receive input of the various and unique characteristics of the agents and introduce a novel encoding method to model the relations among them. Experiments are conducted in the StarCraft multiagent challenge environment, the most famous heterogeneous multiagent simulation, to demonstrate the superior performance of the proposed method compared with the other existing methods in various heterogeneous scenarios. The proposed method in our simulation shows a high win rate and fast convergence speed, proving the superiority of the proposed method considering the heterogeneity of the multiagent system.