A novel reward-shaping-based soft actor–critic for random trajectory tracking of AUVs

Abstract

Current research on autonomous underwater vehicles (AUVs) trajectory tracking mostly focuses on single trajectories, and there is limited research on the generalization of trajectory tracking based on reinforcement learning (RL). This paper introduces a novel RL controller for three-dimensional random trajectory tracking. In this context, a random trajectory includes random obstacles and random reference velocities on the z-axis, and it is designed to improve generalization. The controller integrates value network-based reward shaping (VNRS) with soft actor–critic (SAC). VNRS utilizes a multi-layer perceptron to evaluate the state, which is different from previous work. Simulations demonstrate that VNRS-SAC outperforms SAC in terms of stability and control accuracy. Generalization scenarios, including ocean currents, multiple obstacles, and various trajectories, reveal that the VNRS-SAC controller possesses certain generalization capabilities. Compared with classical S-plane and model predictive control, the VNRS-SAC controller achieves higher control accuracy.