Coordination in Large Multiagent Reinforcement Learning Problems

Abstract

Large distributed systems often require intelligent behavior. Although multiagent reinforcement learning can be applied to such systems, several yet unsolved challenges arise due to the large number of simultaneous learners. Among others, these include exponential growth of state-action spaces and coordination. In this work, we deal with these two issues. Therefore, we consider a subclass of stochastic games called cooperative sequential stage games. With the help of a stateless distributed learning algorithm we solve the problem of growing state-action spaces. Then, we present six different techniques to coordinate action selection during the learning process. We prove a property of the learning algorithm that helps to reduce computational costs of one technique. An experimental analysis in a distributed agent partitioning problem with hundreds of agents reveals that the proposed techniques can lead to higher quality solutions and increase convergence speed compared to the basic approach. Some techniques even outperform a state-of-the-art special purpose approach.