Safe multi-agent deep reinforcement learning for decentralized low-carbon operation in active distribution networks and multi-microgrids

Abstract

Due to fundamental differences in operational entities between distribution networks and microgrids, the equitable allocation of carbon responsibilities remains challenging. Furthermore, achieving real-time, efficient, and secure low-carbon economic dispatch in decentralized multi-entities continues to face obstacles. Therefore, we propose a co-optimization framework for Active Distribution Networks (ADNs) and multi-Microgrids (MMGs) to improve operational efficiency and reduce carbon emissions through adaptive coordination and decision-making. To facilitate decentralized low-carbon decision-making, we introduce the Spatiotemporal Carbon Intensity Equalization Method (STCIEM). This method ensures privacy and fairness by processing local data and equitably distributing carbon responsibilities. Additionally, we propose a non-cooperative optimization strategy that enables entities to optimize their operations independently while considering both economic and environmental interests. To address the challenges of real-time decision-making and the non-convex nature of low-carbon optimization inherent in traditional approaches, we have developed the Enhanced Action Projection Multi-Agent Twin Delayed Deep Deterministic Policy Gradient (EAP-MATD3) algorithm. This algorithm enhances the actor's objective to address the actor-critic mismatch problem, thereby outperforming conventional safe multi-agent deep reinforcement learning methods by generating optimized actions that adhere to physical system constraints. Experiments conducted on the modified IEEE 33-bus network and IEEE 123-bus network demonstrate the superiority of our approach in effectively balancing economic and environmental objectives within complex energy systems.