Deep Reinforcement Learning-Based Collision-Free Navigation for Magnetic Helical Microrobots in Dynamic Environments

Abstract

Magnetic helical microrobots have great potential in biomedical applications due to their ability to access confined and enclosed environments via remote manipulation by magnetic fields. However, achieving collision-free navigation for microrobots in complex and unstructured environments, particularly in highly dynamic settings, remains a challenge. In this paper, we present a novel deep reinforcement learning-based control framework for magnetic helical microrobots, focusing on the tasks of goal-reaching and dynamic obstacle avoidance. To streamline data collection, a specialized training environment capturing essential aspects of navigation for magnetic helical microrobots is devised. The robustness and adaptability of the trained policy are supported using a randomization technique within the training environment. To facilitate seamless integration with real-world magnetic actuation systems, a visual processing algorithm based on OpenCV is devised and incorporated to collect policy observations. Simulations and experiments in various scenarios validate the high robustness and adaptability of the method. The performance assessment revealed a success rate of 99% in navigating the microrobot around 4 dynamic obstacles of comparable speeds and a success rate of 90% in environments with 14 dynamic obstacles. The results indicate the potential for future applications of our method in unstructured, confined, and dynamic living environments.Note to Practitioners—The motivation of this work is to develop a robust and effective control scheme for collision-free navigation of magnetic helical microrobots in dynamic environments. The conventional navigation strategies in dynamic environments mainly include global path planning and local path replanning; thus, highly dynamic environments require frequent updates to the planned path, making it difficult to apply in highly dynamic environments. In this work, a deep reinforcement learning-based control framework is proposed that can guide microrobots through many dynamic obstacles to a series of locations without collisions. The simulation and experimental results validate the efficacy of the proposed control framework and the robustness and adaptability of the trained policy. The proposed control scheme enables better understanding of advanced motion control methods for magnetic microrobots.