Coupled vehicle-signal control based on Stackelberg Game Enabled Multi-agent Reinforcement Learning in mixed traffic environment

Abstract

Related studies on traffic control in partially connected environments either did not consider the collaboration of traffic signal control and vehicular control, or did not consider others’ responsive actions before decision-making in coupled vehicle-signal control. Thus, we propose a Stackelberg Game Enabled Multi-agent Reinforcement Learning (SGMRL) method for coupled vehicle-signal control at an intersection with mixed traffic flow of Connected and Automated Vehicles (CAVs)/Human Driven Vehicles (HDVs). A two-stage framework is applied in SGMRL to learn optimal signal control strategy and CAV platoon strategies in mixed flows of all entrance roads at an intersection. Stackelberg game theory is introduced in SGMRL to make an asynchronous decision-making mechanism. The signal controller is a leader that allocates green times to different phases based on predictions of vehicles’ responsive actions, and CAVs in different directions are followers that form platoons and adjust speeds to adapt to the signal lights decided by the leader. Moreover, CAV platoons in different directions are regarded as agents and form a multi-agent learning framework with the signal controller. Then, an improved Dueling Double Deep Q Network (ID3QN) algorithm is investigated to calculate the Stackelberg equilibrium for the control problem. Experimental results demonstrate that the proposed model effectively reduces the overall waiting time and queue length of all vehicles, in the mixed traffic environment with different CAV penetration rates.