Application of reinforcement learning in planning and operation of new power system towards carbon peaking and neutrality

To mitigate global climate change and ensure a sustainable energy future, China has launched a new energy policy of achieving carbon peaking by 2030 and carbon neutrality by 2060, which sets an ambitious goal of building NPS with high penetration of renewable energy. However, the strong uncertainty, nonlinearity, and intermittency of renewable generation and their power electronics-based control devices are imposing grand challenges for secure and economic planning and operation of the NPS. The performance of traditional methods and tools becomes rather limited under such phenomena. Together with high-fidelity modeling and high-performance simulation techniques, the fast development of AI technology, especially RL, provides a promising way of tackling these critical issues. This paper first provides a comprehensive overview of RL methods that interact with high-fidelity grid simulators to train effective agents for intelligent, model-free decision-making. Secondly, three important applications of RL are reviewed, including device-level control, system-level optimized control, and demand side management, with detailed modeling and procedures of solution explained. Finally, this paper discusses future research efforts for achieving the goals of full absorption of renewable energy, optimized allocation of large-scale energy resources, reliable supply of electricity, and secure and economic operation of the power grid.