Multi-Agent DRL-Controlled Connected and Automated Vehicles in Mixed Traffic With Time Delays

Abstract

The development of intelligent transportation systems (ITS) has attracted significant attention to connected and autonomous vehicles (CAVs). It is urgent to investigate multi-CAV intelligent cruise control solutions in mixed traffic environments. In addition, the impact of platoon dynamics and time delays, induced by shared wireless communications, data processing, and actuation cannot be ignored. This article investigates the development of a multi-agent deep reinforcement learning (MADRL) controller tailored for CAVs operating within mixed and dynamic traffic scenarios that involve time delays. Firstly, the error dynamics in the discrete-time domain for each subplatoon is derived by considering the time-varying delays and leading vehicle states, and then the optimal CAV cruise control problem is formulated. Subsequently, the partially observable Markov game (POMG) is used to construct the multi-agent environment, and then a centralized training decentralized execution (CTDE) algorithm framework is proposed based on the multi-agent deep deterministic policy gradient (MADDPG) method. Finally, the computational complexity and the influence of delay are analyzed. The simulation results illustrate the effectiveness of the proposed intelligent algorithm.