SoftLight: A Maximum Entropy Deep Reinforcement Learning Approach for Intelligent Traffic Signal Control

Abstract

Intelligent traffic signal control plays a crucial role in alleviating traffic congestion. With increasingly available traffic data, there is a trend to use deep reinforcement learning (DRL) techniques for intelligent traffic signal control. However, a majority of existing DRL methods are based on Q-learning, where the optimal solution is always a deterministic policy, so they may fail to adapt to heterogeneous traffic flow and different environment settings. In this paper, we propose a method called SoftLight based on maximum entropy DRL. Through the regularization of maximum entropy, our method learns a stochastic policy that significantly reduces the queue length at the intersection. At the same time, our method keeps the policy as random as possible, which achieves better adaptability to heterogeneous traffic flow. By conducting comprehensive experiments, we demonstrate that our method outperforms existing DRL methods in both phase selection and phase shift settings. We also compare our method with the prevalent maximum entropy DRL method, soft actor-critic (SAC). The results show that our method can find better solutions than SAC under different model designs and hyper-parameters.