Reinforcement learning-based trajectory tracking optimal control of unmanned surface vehicles in narrow water areas.

Abstract

For unmanned surface vehicles (USVs) navigating in narrow water areas in the presence of unknown dynamics and ocean environmental disturbances, this paper develops a reinforcement learning (RL)-based optimal control scheme for the trajectory tracking of USVs under motion state constraints. A nonlinear map is introduced to transform constrained motion state errors into bounded transformed errors, and then the motion state-constrained trajectory tracking problem of USVs is equivalently transformed into a boundedness problem of the transformed errors. Furthermore, an actor-critic framework is developed by utilizing adaptive neural networks (NNs). Within the actor-critic framework, a novel weight update law is designed for the critic NN by combining the gradient descent approach and the concurrent learning technology, thereby relaxing the persistent excitation condition required for adaptive critic NN weight updates. Subsequently, a disturbance compensator is designed and combined with the actor-critic framework to learn the trajectory tracking optimal control law for USVs in the presence of unknown dynamics and disturbances. Finally, theoretical analyses prove that the developed control scheme guarantees the boundedness of all signals in the USV closed-loop trajectory tracking control system, and simulation results show that the developed control scheme can make USVs track the desired trajectory in narrow water areas while reducing the energy consumption by approximately 14.6 % compared with an existing controller.