Knowledge transfer from simple to complex: A safe and efficient reinforcement learning framework for autonomous driving decision-making

Abstract

A safe and efficient decision-making system is crucial for autonomous vehicles. However, the complexity of driving environments often limits the effectiveness of many rule-based and machine learning approaches. Reinforcement learning (RL), with its robust self-learning capabilities and adaptability to diverse environments, offers a promising solution. Despite this, concerns about safety and efficiency during the training phase have hindered its widespread adoption. To address these challenges, we propose a novel RL framework, Simple to Complex Collaborative Decision (S2CD), based on the Teacher–Student Framework (TSF) to facilitate safe and efficient knowledge transfer. In this approach, the teacher model is first trained rapidly in a lightweight simulation environment. During the training of the student model in more complex environments, the teacher evaluates the student’s selected actions to prevent suboptimal behavior. Besides, to enhance performance further, we introduce an RL algorithm called Adaptive Clipping Proximal Policy Optimization Plus (ACPPO+), which combines samples from both teacher and student policies while utilizing dynamic clipping strategies based on sample importance. This approach improves sample efficiency and mitigates data imbalance. Additionally, Kullback–Leibler (KL) divergence is employed as a policy constraint to accelerate the student’s learning process. A gradual weaning strategy is then used to enable the student to explore independently, overcoming the limitations of the teacher. Moreover, to provide model interpretability, the Layer-wise Relevance Propagation (LRP) technique is applied. Simulation experiments conducted in highway lane-change scenarios demonstrate that S2CD significantly enhances training efficiency and safety while reducing training costs. Even when guided by suboptimal teachers, the student consistently outperforms expectations, showcasing the robustness and effectiveness of the S2CD framework.