Q-Learning Acceleration via State-Space Partitioning

Abstract

One of the biggest obstacles of Reinforcement Learning (RL) is its slow convergence rate in large state spaces or with sparse rewards. It has been shown that single-agent RL can be accelerated within a cooperative multi-agent scenario with information sharing, however the speedup depends on how well the agents' information can be used together. We demonstrate in this paper that state-space partitioning among agents can be realized by reward design without hard coded rules. The partitioning-associated reward directs agents to focus on different partitions and thus share information more efficiently. This approach has two advantages: (1) agents' actions are not diminished and remain relatively independent from one another; (2) it can be used to accelerate learning in both structured state domains (where partitions can be pre-determined) and arbitrarily-structured state domains (where partitions may be developed dynamically by agent teams as they explore the environment). Finally, we validate the method's efficacy by comparing it to previous related work in a simplified soccer domain.