Jonghyeok Park , Jongsoo Lee , Jangwon Kim , Soohee Han
{"title":"Overcoming intermittent instability in reinforcement learning via gradient norm preservation","authors":"Jonghyeok Park , Jongsoo Lee , Jangwon Kim , Soohee Han","doi":"10.1016/j.ins.2025.122081","DOIUrl":null,"url":null,"abstract":"<div><div>Instability is a critical challenge in online reinforcement learning (RL), particularly in robotics. While existing RL methods primarily address this intractable instability through algorithmic modifications or training strategies, the role of optimization techniques remains largely underexplored. This paper investigates intermittent instability in RL training, which hinders accurate value learning, and proposes a novel optimization approach: Gradient Norm Preservation (GNP). Our analysis identifies irregular gradient spikes, caused by high-reward data during exploration, as a key source of instability. These spikes are mathematically quantified, and the optimizer's learning rate is dynamically adjusted to preserve initial gradient norms, mitigating their impact on value learning. Experiments across diverse environments demonstrate that integrating GNP into RL algorithms significantly improves stability, with notable gains in training performance across several environments.</div></div>","PeriodicalId":51063,"journal":{"name":"Information Sciences","volume":"709 ","pages":"Article 122081"},"PeriodicalIF":8.1000,"publicationDate":"2025-03-13","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Information Sciences","FirstCategoryId":"94","ListUrlMain":"https://www.sciencedirect.com/science/article/pii/S0020025525002130","RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"0","JCRName":"COMPUTER SCIENCE, INFORMATION SYSTEMS","Score":null,"Total":0}
引用次数: 0
Abstract
Instability is a critical challenge in online reinforcement learning (RL), particularly in robotics. While existing RL methods primarily address this intractable instability through algorithmic modifications or training strategies, the role of optimization techniques remains largely underexplored. This paper investigates intermittent instability in RL training, which hinders accurate value learning, and proposes a novel optimization approach: Gradient Norm Preservation (GNP). Our analysis identifies irregular gradient spikes, caused by high-reward data during exploration, as a key source of instability. These spikes are mathematically quantified, and the optimizer's learning rate is dynamically adjusted to preserve initial gradient norms, mitigating their impact on value learning. Experiments across diverse environments demonstrate that integrating GNP into RL algorithms significantly improves stability, with notable gains in training performance across several environments.
期刊介绍:
Informatics and Computer Science Intelligent Systems Applications is an esteemed international journal that focuses on publishing original and creative research findings in the field of information sciences. We also feature a limited number of timely tutorial and surveying contributions.
Our journal aims to cater to a diverse audience, including researchers, developers, managers, strategic planners, graduate students, and anyone interested in staying up-to-date with cutting-edge research in information science, knowledge engineering, and intelligent systems. While readers are expected to share a common interest in information science, they come from varying backgrounds such as engineering, mathematics, statistics, physics, computer science, cell biology, molecular biology, management science, cognitive science, neurobiology, behavioral sciences, and biochemistry.