Attention-Enhanced Multi-Agent Reinforcement Learning Against Observation Perturbations for Distributed Volt-VAR Control

Abstract

The cloud-edge collaboration architecture has been widely adopted for distributed Volt-VAR control (VVC) problems in active distribution networks (ADNs). To alleviate the computation and communication burden on edge sides, centralized training & decentralized execution (CTDE) based multi-agent reinforcement learning methods have been proposed. However, the performance of these methods relies heavily on the agents’ coordination mechanism and accurate observations. Given access to a vast amount of distributed energy resources, it becomes increasingly challenging to achieve efficient coordination within CTDE framework. Furthermore, the agents’ observations always involve perturbations such as measurement noises and even cyber-attacks in real-world ADNs, which can significantly degrade the distributed VVC performance and may cause severe security issues. In this paper, we propose an attention-enhanced multi-agent reinforcement learning method to address observation perturbations for distributed VVC. In our proposed method, a mix network on the cloud platform with an agent-level attention mechanism is used to approximate the global reward, successfully capturing the intercorrelations between agents and achieving excellent coordination. A novel robust regularizer is also designed to enhance the agents’ robustness facing the observation perturbations, which greatly improves the applicability of reinforcement learning methods. Numerical simulations on IEEE test cases with real-world data demonstrate the effectiveness and robustness of our proposed method.