Enabling microrobotic chemotaxis via reset-free hierarchical reinforcement learning

Abstract

Microorganisms have evolved diverse strategies to propel in viscous fluids, navigate complex environments, and exhibit taxis in response to stimuli. This has inspired the development of synthetic microrobots, where machine learning (ML) is playing an increasingly important role. Can ML endow these robots with intelligence resembling that developed by their natural counterparts over evolutionary timelines? Here, we demonstrate chemotactic navigation of a multi-link articulated microrobot using two-level hierarchical reinforcement learning (RL). The lower-level RL allows the robot -- featuring either a chain or ring topology -- to acquire topology-specific swimming gaits: wave propagation characteristic of flagella or body oscillation akin to an ameboid. Such flagellar and ameboid microswimmers, further enabled by the higher-level RL, accomplish chemotactic navigation in prototypical biologically-relevant scenarios that feature conflicting chemoattractants, pursuing a swimming bacterial mimic, steering in vortical flows, and squeezing through tight constrictions. Additionally, we achieve reset-free, partially observable RL, where the robot observes only its joint angles and local scalar quantities. This advancement illuminates solutions for overcoming the persistent challenges of manual resets and partial observability in real-world microrobotic RL.