AdapSafe2: Prior-Free Safe-Certified Reinforcement Learning for Multi-Area Frequency Control

Abstract

Safe Reinforcement Learning (RL) has been widely investigated to conduct power systems frequency control under high renewable energy resources penetration. Nevertheless, existing safe RL-based frequency control methods still face two fundamental challenges to achieving safety guarantees: (1) operating in non-stationary environments without the prior knowledge of the system parameters and (2) simultaneously satisfying high-dimensional and time-varying safety constraints in multi-area cases. To this end, this paper proposes a prior-free reinforcement learning-based frequency control method with guaranteed safety for multi-area power systems named AdapSafe2. To tackle Challenge (1), a meta-based environmental learning algorithm is developed to automatically capture and rapidly adapts to non-stationary system parameters without relying on a predefined nominal model. Furthermore, a meta-RL framework is established to achieve a self-adaptive frequency control strategy without prior knowledge. Moreover, for Challenge (2), a novel safety-critic network and a safe-certified compensator based on the control barrier function are designed to identify time-varying safety constraints. Leveraging risk assessments from the safety-critic network, the compensator performs dynamic safety compensations only for areas with risk, thereby enhancing the efficiency of solving under high-dimensional safety constraints. Numerical simulations conducted under 2-Area and 3-Area wind-aggregated low-inertia power systems demonstrate that the proposed AdapSafe2 can outperform the state-of-the-art approaches while effectively satisfying the dynamic safety constraints.