Delayed Dynamic Model Scheduled Reinforcement Learning With Time-Varying Delays for Robotic Control

Abstract

Reinforcement learning (RL) typically presupposes instantaneous agent-environment interactions, but in real-world scenarios such as robotic control, overlooking observation delays can significantly impair performance. While existing studies consider stationary, known delays, real-world applications frequently encounter unpredictable delay variations. To address this problem, this letter presents a novel algorithm for scheduling delayed dynamic models. Specifically, We propose using multiple truncated delay distributions to effectively model time-varying delays, with each distribution tailored to learn a specific delayed dynamic model. These models map delayed observations and historical actions to the current state, integrating seamlessly with existing RL algorithms to facilitate optimal decision-making. Since the delay is unknown to the agent, we propose an effective delay estimation method to detect delay and their corresponding distributions in real-time, thereby adaptively selecting the most appropriate delayed dynamic model to manage delays. To reduce instability caused by abrupt changes in delay distribution and enhance responsiveness to such variations, we apply Bayesian online changepoint detection to enable rapid sensing of alterations in the delay distribution within a finite number of time-steps. To the best of our knowledge, our approach is the first effective solution to the non-stationary time-varying delay problem in RL. Empirical results demonstrate the robust performance of our method in scenarios characterized by non-stationary observation delays, highlighting its strong potential for robotic control applications.