Estimating Lyapunov Region of Attraction for Robust Model-Based Reinforcement Learning USV

Abstract

This article addresses the robustness of unmanned surface vehicles (USV) using model-based reinforcement learning (MBRL). A novel MBRL approach, Lyapunov probabilistic model predictive control (LPMPC) is proposed to simultaneously learn both the probabilistic model of a USV and its corresponding estimated Lyapunov region of attraction (ROA) under one reinforcement learning framework. Unlike the existing MBRL USV systems with less consideration of robustness and safety, our method naturally learns a general indicator of system stability based on the probabilistic model’s belief and employs it to guide its policy. Evaluated by different navigation tasks in a simulation driven by real boat data, LPMPC demonstrated significant advantages in both control robustness and task completion against various levels of environmental disturbances compared with the baseline approach without Lyapunov ROA’s guidance. Note to Practitioners—Modelling the system stability without human prior knowledge is challenging in the domain of USV. This work proposed a data-driven method to iteratively learn a task-relevant stability model of USV in a probabilistic view. Based on the evaluation of a real boat data-driven simulation, the learned stability model contributed to superior driving skills in different USV scenarios by properly indicating and avoiding potentially risky states. In future research, we plan to expand the definition of risks in different tasks, such as loss of control, overlarge sway, and excessive energy consumption and investigate the proposed approach in real-world USV.