Learning to play Football using Distributional Reinforcement Learning and Depthwise separable convolution feature extraction

Abstract

In recent years we have seen a huge surge in benchmark tasks for Deep Reinforcement learning algorithms and a tremendous growth in the field of reinforcement learning itself but oftentimes the stochasticity and real time decision making of real world strategic and competitive games and also the choice between a multitude of actions are not mirrored in the environments used for RL agents. To address this issue Google released Gfootball, a football game engine based environment which was popularized by Manchester City FC sponsoring a Kaggle competition but the majority methods revolved around Rule based RL agent, imitation learning, reward modifications etc. and the pure reinforcement learning included feature extractors which had parallel neural networks on costly hardware. We propose a much simpler method involving depthwise separable convolutions as the base feature extractor which yields competitive results across a lot of benchmarks in very few episodes compared to the original paper. We also used Quantile regression DQN due to highly stochastic nature of the environment to exploit the quantiles of the return distribution to improve performance.