Robust Multiagent Reinforcement Learning toward Coordinated Decision-Making of Automated Vehicles

Abstract

Automated driving is essential for developing and deploying intelligent transportation systems. However, unavoidable sensor noises or perception errors may cause an automated vehicle to adopt suboptimal driving policies or even lead to catastrophic failures. Additionally, the automated driving longitudinal and lateral decision-making behaviors (e.g., driving speed and lane changing decisions) are coupled, that is, when one of them is perturbed by unknown external disturbances, it causes changes or even performance degradation in the other. The presence of both challenges significantly curtails the potential of automated driving. Here, to coordinate the longitudinal and lateral driving decisions of an automated vehicle while ensuring policy robustness against observational uncertainties, we propose a novel robust coordinated decision-making technique via robust multiagent reinforcement learning. Specifically, the automated driving longitudinal and lateral decisions under observational perturbations are modeled as a constrained robust multiagent Markov decision process. Meanwhile, a nonlinear constraint setting with Kullback–Leibler divergence is developed to keep the variation of the driving policy perturbed by stochastic perturbations within bounds. Additionally, a robust multiagent policy optimization approach is proposed to approximate the optimal robust coordinated driving policy. Finally, we evaluate the proposed robust coordinated decision-making method in three highway scenarios with different traffic densities. Quantitatively, in the absence of noises, the proposed method achieves an approximate average enhancement of 25.58% in traffic efficiency and 91.31% in safety compared to all baselines across the three scenarios. In the presence of noises, our technique improves traffic efficiency and safety by an approximate average of 30.81% and 81.02% compared to all baselines in the three scenarios, respectively. The results demonstrate that the proposed approach is capable of improving automated driving performance and ensuring policy robustness against observational uncertainties.