HO2RL: A Novel Hybrid Offline-and-Online Reinforcement Learning Method for Active Pantograph Control

Abstract

The pantograph–catenary system (PCS) is vital for high-speed trains to collect electrical power, where the contact force fluctuation seriously reduces the current collection quality, increases maintenance costs, and affects operation safety. Reinforcement learning (RL) is an attractive approach for learning active pantograph control policy by trial and error. However, the traditional RL methods suffer significant performance degradation or collapse when deployed to the real world due to the huge sim-real gap. We propose a hybrid offline-and-online reinforcement learning (HO2RL) algorithm to solve active pantograph control tasks, which elegantly combines RL policy pretraining with offline transitions and performance enhancement with online data collection. The proposed algorithm provides generalized pretrained models by learning effective behavior policy from offline experiences and then performs multidomain adaptation by online performance improvement with dynamics-aware policy evaluation. Experimental results demonstrate that the HO2RL algorithm efficiently learns from large and diverse static datasets and enables steady performance improvement by fine-tuning with online interactions. The proposed method solves active pantograph control tasks in various operation scenarios and demonstrates SOTA performance on the PCS standard benchmark.