Hierarchical Gait Generation for Modular Robots Using Deep Reinforcement Learning

Modular robots have the ability to perform versatile locomotion with a high diversity of morphologies. However, designing robust locomotion gaits for arbitrary robot morphologies remains exceptionally challenging. In this paper, a two-level hierarchical locomotion framework is presented for addressing modular robot locomotion tasks. The framework combines a central pattern generator controller (CPG) with a neural network trained by deep reinforcement learning. First, the low-level CPG controllers are learned by offline optimization and generate robust straight walking gaits. Second, a high-level neural network is then learned using deep reinforcement learning via trial-and-errors. The high-level learned controller can modulate the low-level CPG parameters based on online inputs including robot states and user commands. Simulation experiments are employed on a 3D modular robot. The results show that the proposed method achieves better overall performance than the baseline methods on different locomotion skills including straight walking, velocity tracking, and circular turning. Simulation results confirm the effectiveness and robustness of the proposed method.