Economical and Reliable Energy Management for Networked Microgrids in a Multi-Agent Collaborative Manner

Abstract

Reliability is a fundamental requirement of power systems. However, uncertainties from renewable energy generators and demand loads bring challenges to the economical and reliable operation of power distribution networks. This paper focuses on an energy management problem for networked microgrid systems (NMSs), aiming at establishing energy management policies for individual microgrids within NMSs to enhance the long-term economy and stability of the entire system. The key challenge is that each microgrid has independent decision-making capability and how to collaborate all microgrids’ decisions in a distributed multi-agent manner such that the entire system goal is optimized. We introduce the mean and variance of the exchanged power between the NMS and the main grid as the indicators of NMS operation profits and power fluctuations, respectively. The energy management problem is formulated as a mean-variance team stochastic game (MV-TSG) model. Because the variance metric is neither additive nor Markovian in a dynamic setting, the dynamic programming principle does not hold for this problem. We analyze the MV-TSG from a view of sensitivity-based optimization and propose a multi-agent policy iteration method by introducing a sequential update scheme. Furthermore, to tackle MV-TSGs with larger scales or unknown environmental parameters, we extend the idea of trust region optimization and develop a novel multi-agent deep reinforcement learning algorithm. The performance of our approaches is verified through numerical experiments using a real dataset in power distribution networks. Note to Practitioners—Energy management is essential for economical and reliable operation of networked microgrid systems (NMSs). NMSs exchange power with the main grid, where positive exchanged power indicates that the NMS is selling power to the main grid and generating profits, while negative exchanged power indicates that the NMS is purchasing power and incurring costs. The long-term average exchanged power between the NMS and the main grid reflects the economical operation of NMSs. However, fluctuations in exchanged power can lead to power quality issues such as frequency deviation and voltage flicker, which affect the reliability of NMS operations. In this paper, we study the economical and reliable operation of NMSs with distributed energy management policies, formulating the problem as a mean-variance team stochastic game (MV-TSG). We propose a multi-agent policy iteration method to address MV-TSGs. We further develop a multi-agent deep reinforcement learning algorithm to tackle MV-TSGs with large scales and in scenarios where environmental models are unknown. Experiments utilizing real data demonstrate that our algorithms effectively promote collaboration among microgrids to control exchanged power. As the weight coefficient increases, power fluctuations are reduced, achieving power smoothing and peak shaving simultaneously in long-term scheduling. Consequently, our algorithms can facilitate the trade-off between economy and reliability in NMS operations. These results provide valuable insights into the energy management challenges of NMSs in power distribution networks.