A load frequency control strategy based on double deep Q-network and upper confidence bound algorithm of multi-area interconnected power systems

Abstract

The reinforcement learning (RL)-based generation control strategies have been widely studied to address the limited adaptability of traditional automatic generation control (AGC) strategies to the load disturbance problem resulting from heterogeneous energy sources. To improve the control accuracy of the RL-based strategy in load frequency control (LFC), a double deep Q-network combined with an upper confidence bound (DDQN-UCB)-based strategy is designed to solve the problem of agent decision-making in a nonlinear environment. Firstly, the area control error (ACE) and control performance standard 1 (CPS1) of the LFC power system are considered in the design of the RL reward function. Secondly, the actual and estimated Q-values are calculated using the Q-network and the target Q-network combined with the reward value. Thirdly, the deviation loss of the two Q-values is calculated, and the network is updated based on the loss value using gradient descent. Finally, the UCB algorithm is introduced to equalize the frequency of being selected for each action during the random exploration of the actions, and the agent uses the greedy algorithm in combination with the UCB algorithm to select a power-compensated control action to send to the environment. In this paper, the IEEE multi-area LFC power system is used as an experimental validation model. A comparison of the proposed RL control algorithm with five other algorithms revealed that the pre-learning convergence accuracy was improved by 57.5%. Furthermore, the LFC effectiveness test demonstrated that the DDQN-UCB control strategy enhances LFC accuracy while simultaneously stabilizing the power exchange of the inter-area tie-line to within 1.8972 MW, thereby maintaining the stability of the power system.