Q-learning of complex behaviours on a six-legged walking machine

We present work on a six-legged walking machine that uses a hierarchical version of Q-learning (HQL) to learn both the elementary swing and stance movements of individual legs as well as the overall coordination scheme to perform forward movements. The architecture consists of a hierarchy of local controllers implemented in layers. The lowest layer consists of control modules performing elementary actions, like moving a leg up, down, left or right to achieve the elementary swing and stance motions for individual legs. The next level consists of controllers that learn to perform more complex tasks like forward movement by using the previously learned, lower level modules. On the third the highest layer in the architecture presented here the previously learned complex movements are themselves reused to achieve goals in the environment using external sensory input. The work is related to similar, although simulation-based, work by Lin (1993) on hierarchical reinforcement learning and Singh (1994) on compositional Q-learning. We report on the HQL architecture as well as on its implementation on the walking machine SIR ARTHUR. Results from experiments carried out on the real robot are reported to show the applicability of the HQL approach to real world robot problems.