Overcoming intermittent instability in reinforcement learning via gradient norm preservation

Abstract

Instability is a critical challenge in online reinforcement learning (RL), particularly in robotics. While existing RL methods primarily address this intractable instability through algorithmic modifications or training strategies, the role of optimization techniques remains largely underexplored. This paper investigates intermittent instability in RL training, which hinders accurate value learning, and proposes a novel optimization approach: Gradient Norm Preservation (GNP). Our analysis identifies irregular gradient spikes, caused by high-reward data during exploration, as a key source of instability. These spikes are mathematically quantified, and the optimizer's learning rate is dynamically adjusted to preserve initial gradient norms, mitigating their impact on value learning. Experiments across diverse environments demonstrate that integrating GNP into RL algorithms significantly improves stability, with notable gains in training performance across several environments.