A new noise network and gradient parallelisation-based asynchronous advantage actor-critic algorithm

Abstract

Asynchronous advantage actor-critic (A3C) algorithm is a commonly used policy optimization algorithm in reinforcement learning, in which asynchronous is parallel interactive sampling and training, and advantage is a sampling multi-step reward estimation method for computing weights. In order to address the problem of low efficiency and insufficient convergence caused by the traditional heuristic exploration of A3C algorithm in reinforcement learning, an improved A3C algorithm is proposed in this paper. In this algorithm, a noise network function, which updates the noise tensor in an explicit way is constructed to train the agent. Generalised advantage estimation (GAE) is also adopted to describe the dominance function. Finally, a new mean gradient parallelisation method is designed to update the parameters in both the primary and secondary networks by summing and averaging the gradients passed from all the sub-processes to the main process. Simulation experiments were conducted in a gym environment using the PyTorch Agent Net (PTAN) advanced reinforcement learning library, and the results show that the method enables the agent to complete the learning training faster and its convergence during the training process is better. The improved A3C algorithm has a better performance than the original algorithm, which can provide new ideas for subsequent research on reinforcement learning algorithms.