Efficient navigation in vortical flows based on reinforcement learning and flow field prediction

Abstract

In this paper, we address the navigation problem of autonomous agents in complex, time-varying flow fields using Deep Reinforcement Learning (DRL). Specifically, the Proximal Policy Optimization (PPO) algorithm is used to solve Zermelo’s problem for point-to-point navigation tasks. The challenge of navigation in this article arises from the fact that the agent’s movement speed is slower than the surrounding flow velocity, requiring the agent to adapt to the flow dynamics rather than simply counteracting it. We propose the Look-Ahead State-Space (LASS) method as a novel approach to enhance navigation performance by enabling the agent to anticipate future states, which incorporate information from either true or predicted flow fields. A long short-term memory network combined with a transposed convolutional network is used to predict the future flow dynamics based solely on historical sensory data from the agent. Our results demonstrate that the LASS strategy improves the agent’s adaptability and significantly improves navigation success rates, even in dynamic environments. Additionally, we compare the PPO-based navigation method with an optimal control planner, revealing that while optimal control achieves marginally faster travel times, the DRL-based approach offers significant advantages in computational efficiency, making it more suitable for real-time applications.