Intelligent Set Speed Estimation for Vehicle Longitudinal Control with Deep Reinforcement Learning

Besides the goal of reducing driving tasks, modern longitudinal control systems also aim to improve fuel efficiency and driver comfort. Most of the vehicles use Adaptive Cruise Control (ACC) systems that track constant set speeds and set headways which makes the trajectory of the vehicle in headway mode highly dependent on the trajectory of a preceding vehicle. Hence, this might lead to increased consumptions in dense traffic situations or when the leader has a less careful driving style. In this work, a method based on Deep Reinforcement Learning (DRL) is presented that finds a control strategy by estimating an intelligent variable set speed based on the system state. Additional control objectives, such as minimizing consumption, are considered explicitly through the feedback in a reward function. A DRL framework is set up that enables the training of a neural set speed estimator for vehicle longitudinal control in a simulative environment. The Deep Deterministic Policy Gradient algorithm is used for the training of the agent. Training is carried out on a simple test track to teach the basic concepts of the control objective to the DRL agent. The learned behavior is then examined in a more complex, stochastic microscopic traffic simulation of the city center of Darmstadt and is compared to a conventional ACC algorithm. The analysis shows that the DRL controller is capable of finding fuel efficient trajectories which are less dependent on the preceding vehicle and is able to generalize to more complex traffic environments, but still shows some unexpected behavior in certain situations. The combination of DRL and conventional models to build up on the existing engineering knowledge is therefore expected to yield promising results in the future.