Charging scheduling strategy for electric vehicles in residential areas based on offline reinforcement learning

Abstract

As the number of electric vehicles(EVs) increases, reinforcement learning(RL) faces more challenges in EV charging scheduling. Online RL requires lots of interaction with the environment and trial and error, which may lead to high costs and potential risks. In addition, the large-scale application of EVs causes curse of dimensionality in RL. In response to these problems, this work constructed a residential area microgrid model that comprehensively considered the nonlinear charging models of different types of EVs and the vehicle-to-grid (V2G) mode. The charging scheduling problem is represented as a Constrained Markov Decision Process (CMDP), employing a model-free RL framework to proficiently address uncertainties. In response to the curse of dimensionality problem, this paper designs a charging strategy, and divides EVs into different sets according to their statuses. The agent transmits control signals to the sets, thereby efficiently reducing the dimension of the action space. Subsequently, the Lagrangian-BCQ algorithm is trained using the offline data set, the charging strategy based on the Lagrangian-BCQ algorithm is employed to address the CMDP, with the incorporation of a safety filter to guarantee compliance with stringent constraints. Through numerical simulation experiments, the effectiveness of the strategy proposed in this work was verified.