DWAS-RL: A safety-efficiency balanced reinforcement learning approach for path planning of Unmanned Surface Vehicles in complex marine environments

Abstract

Navigating autonomous surface vehicles in dynamic marine environments, where uncertainties and disturbances like static or moving obstacles, ocean currents, and waves abound, poses a formidable challenge. Recent advancements in Deep Reinforcement Learning (DRL) have shown promising results in terms of adaptivity and timeliness through interaction with the environment. However, effectively addressing zero safety violations while achieving sample efficiency remains a dual challenge in practical applications. In this paper, we strive to ensure both safety and learning efficiency by combining the advantages of the Dynamic Window Approach (DWA) and safe reinforcement learning. First, a customized simulator for diverse marine conditions is developed, where various types of marine scenarios and algorithms are trained and testified. Then, the problem is formulated as a constrained Markov decision process and the DWA-based safe RL (DWAS-RL) approach is proposed. Specifically, to guarantee safety in the exploration process, we utilize DWA to observe and generate prudent actions by predicting potential near-future hazards, then utilize the safe RL framework for exploration and training. To improve sample efficiency, the technique called Hindsight Experience Replay is utilized to accelerate the training process. Simulation experiments demonstrate the effectiveness of our approach on the metrics of kinematics performance, safety and sample efficiency compared to the state-of-the-art DRL algorithms. These findings highlight the robustness and superiority of our approach, suggesting that our approach holds promise for addressing challenges in complex marine environments.