XLight: An interpretable multi-agent reinforcement learning approach for traffic signal control

Abstract

Recently, deep reinforcement learning (DRL)-based traffic signal control (TSC) methods have garnered significant attention among researchers, achieving substantial progress. However, current research often focuses on performance improvement, neglecting interpretability. DRL-based TSC methods often face challenges in interpretability. This limitation poses significant obstacles to practical deployment, given the liability and regulatory constraints faced by governmental authorities responsible for traffic management and control. On the other hand, interpretable RL-based TSC methods offer greater flexibility to meet specific requirements. For instance, prioritizing the clearance of vehicles in a particular movement can be easily achieved by assigning higher weights to the state variables associated with that movement. To address this issue, we propose Xlight, an interpretable multi-agent reinforcement learning (MARL) approach for TSC, which enhances interpretability in three key aspects: (a) meticulously designing and selecting the state space, action space, and reward function. Especially, we propose an interpretable reward function for network-wide TSC and prove that maximizing this reward is equivalent to minimizing the average travel time (ATT) in the road network; (b) introducing more practical regulatable (i.e., interpretable) functions as TSC controllers; and (c) employing maximum entropy policy optimization, which simultaneously enhances interpretability and improves transferability. Next, to better align with practical applications of network-wide TSC, we propose several interpretable MARL-based methods. Among these, Multi-Agent Regulatable Soft Actor-Critic (MARSAC) not only possesses interpretability but also achieves superior performance. Finally, comprehensive experiments conducted across various TSC scenarios, including isolated intersection, synthetic network-wide intersections, and real-world network-wide intersections, demonstrate the effectiveness. For example, in terms of the ATT metric, our proposed method achieves improvements of 9.55%, 34.17%, 3.98%, and 42.93% compared to the Actuated Traffic Signal Control (ATSC) across a synthetic road network and 3 real-world road networks. Furthermore, in the synthetic network, our method demonstrates improvements of 4.04% and 3.21% in the Safety Score and Fuel Consumption metrics, respectively, when compared to the ATSC.