Efficient Real2Sim2Real of Continuum Robots Using Deep Reinforcement Learning With Koopman Operator

Abstract

Accurate control of continuum robots is challenging, especially in the presence of external disturbances. To address this issue, reinforcement learning (RL) has been increasingly investigated in continuum robot control due to its online policy updating capability. However, the gap in Sim2Real transfer caused by inaccurate modeling results in RL implementation a dilemma. In this article, we propose a safety-critical Real2Sim2Real online RL framework, where the Real2Sim transfer is first achieved by the identification of a continuum robot using the Koopman operator. We further improve the training efficiency by introducing an imperfect demonstration into the RL framework. The offline policy is trained in simulations and then tested on a real continuum robot platform. During tests, the tracking performance is influenced by the hysteresis effect that cannot be captured by the Koopman operator. This results in a millimeter-level tracking root mean square error (RMSE). To address this issue, we online update the policy as well as the model, and the RMSE of the online controller outperforms the offline controller by 89.16% in free space and 85.70% under external payload, respectively.