Robust Reinforcement Learning for Decision Making Under Uncertainty in Electricity Markets

Abstract

Reinforcement learning (RL) is a powerful tool for market agents solving decision-making problems in electricity markets. Vanilla RL enables agents to learn optimal policies in dynamic and uncertain market environments via trial and error. However, uncertainties in state transitions are often treated as exogenous state features with statistical errors. This approach can result in policies that are sensitive to perturbations of these uncertainties, potentially leading to performance degradation. This sensitivity is particularly critical in electricity markets, where the penetration of renewable energy and demand variability are increasing. To address this issue, this paper proposes a robust adversarial RL algorithm aimed at learning a robust optimal policy that accounts for market uncertainties in state transitions to systematically mitigate sensitivity to perturbations in uncertain environments. Specifically, we leverage the uncertainty set regularizer technique to define uncertainty sets within the parametric space of state transitions. Furthermore, we introduce a novel adversarial approach to generate unknown uncertainty sets using the value function as a basis. We finally conduct a comprehensive assessment of the robust adversarial RL algorithm across three electricity market applications: strategic bidding, retail pricing, and peer-to-peer energy trading, demonstrating significant improvements in robustness performance against various uncertainties.