Continuous reinforcement learning from human demonstrations with integrated experience replay for autonomous driving

Abstract

As a promising subfield of machine learning, Reinforcement Learning (RL) has drawn increasing attention among the academia as well as the public. However, the practical application of RL is still restricted by a variety of reasons. The two most significant challenges of RL are the large exploration domain and the difficulty to converge. Integrating RL with human expertise is technically an interesting way to accelerate the exploration and increase the stability. In this work, we propose a continuous reinforcement learning method which integrates Deep Deterministic Policy Gradient (DDPG) with human demonstrations. The proposed method uses a combined loss function for updating the actor and critic networks. In addition, the experience replay buffer is also drawn from different transition data samples to make the learning more stable. The proposed method is tested with a popular RL task, i.e. the autonomous driving, by simulations with TORCS environment. Experimental results not only show the effectiveness of our method in improving the learning stability, but also manifest the potential capability of our method in mastering human preferences.