Lane-changing policy offline reinforcement learning of autonomous vehicles based on BEAR algorithm with support set constraints

Abstract

Imitation learning struggles to learn an optimal policy from datasets containing both expert and non-expert samples due to its inability to discern the quality differences between these samples. Furthermore, standard online reinforcement learning (RL) methodologies face significant exploration costs and safety risks during environmental interactions. Addressing these challenges, this study develops a lane-changing model for autonomous vehicles using the bootstrapping error accumulation reduction (BEAR) algorithm. The model initially examines the distributional shifts between behavioral and target policies in offline RL. It then incorporates the BEAR algorithm, enhanced with support set constraints, to mitigate this issue. The study subsequently proposes a lane-changing policy learning method based on the BEAR algorithm in offline RL. This method involves designing the state space, action set, and reward function. The reward function is tailored to guide the autonomous vehicle in executing lane changes while balancing safety, ride comfort, and traffic efficiency. In the final stage, the lane-changing policy is learned using a dataset of both expert and non-expert samples. Test results indicate that the lane-changing policy developed through this method shows higher success rates and safety levels compared to policies derived via imitation learning.