SAR: generalization of physiological agility and dexterity via synergistic action representation

Abstract

Learning effective continuous control policies in high-dimensional systems, including musculoskeletal agents, remains a significant challenge. Over the course of biological evolution, organisms have developed robust mechanisms for overcoming this complexity to learn highly sophisticated strategies for motor control. What accounts for this robust behavioral flexibility? Modular control via muscle synergies, i.e. coordinated muscle co-contractions, is considered to be one putative mechanism that enables organisms to learn muscle control in a simplified and generalizable action space. Drawing inspiration from this evolved motor control strategy, we use physiologically accurate human hand and leg models as a testbed for determining the extent to which a Synergistic Action Representation (SAR) acquired from simpler tasks facilitates learning and generalization on more complex tasks. We find in both cases that SAR-exploiting policies significantly outperform end-to-end reinforcement learning. Policies trained with SAR were able to achieve robust locomotion on a diverse set of terrains (e.g., stairs, hills) with state-of-the-art sample efficiency (4 M total steps), while baseline approaches failed to learn any meaningful behaviors under the same training regime. Additionally, policies trained with SAR on in-hand 100-object manipulation task significantly outperformed (>70% success) baseline approaches (<20% success). Both SAR-exploiting policies were also found to generalize zero-shot to out-of-domain environmental conditions, while policies that did not adopt SAR failed to generalize. Finally, using a simulated robotic hand and humanoid agent, we establish the generality of SAR on broader high-dimensional control problems, solving tasks with greatly improved sample efficiency. To the best of our knowledge, this investigation is the first of its kind to present an end-to-end pipeline for discovering synergies and using this representation to learn high-dimensional continuous control across a wide diversity of tasks. Project website:https://sites.google.com/view/sar-rl