Designing Optimal Arbitrage Policies for Distributed Energy Systems in Building Clusters Using Reinforcement Learning

Abstract

In the wake of increasing proliferation of renewable energy and distributed energy resources (DERs), grid designers and operators alike are faced with several emerging challenges in curbing allocative grid inefficiencies and maintaining operational stability. One such challenge relates to the increased price volatility within real-time electricity markets, a result of the inherent intermittency of renewable energy. With this challenge, however, comes heightened economic interest in exploiting the arbitrage potential of price volatility towards demand-side energy cost savings. To this end, this paper aims to maximize the arbitrage value of electricity through the optimal design of control strategies for DERs. Formulated as an arbitrage maximization problem using design optimization, and solved using reinforcement learning, the proposed approach is applied towards shared DERs within multi-building residential clusters. We demonstrate its feasibility across three unique building cluster demand profiles, observing notable energy cost reductions over baseline values. This highlights a capability for generalized learning across multiple building clusters and the ability to design efficient arbitrage policies towards energy cost minimization. Finally, the approach is shown to be computationally tractable, designing efficient strategies in approximately 5 hours of training over a simulation time horizon of 1 month.