Data-driven pre-training framework for reinforcement learning of air-source heat pump (ASHP) systems based on historical data in office buildings: Field validation

Abstract

Reinforcement Learning (RL) has demonstrated potential for optimal control of Heating, Ventilation, and Air Conditioning (HVAC) systems. Current research on RL in HVAC systems control is limited to simulation studies, with few real-world deployments that have minimal focus on supply-side optimization, along with a reliance on building simulation tools for pre-training. This paper proposes a practical data-driven pre-training framework for Air-Source Heat Pump (ASHP) system. The framework integrates data-driven models based on real-world historical data for load forecasting, equipment energy consumption, and heat transfer. As a case study, two classic value-based reinforcement learning agents, Q-learning and Deep Q-Network (DQN), were selected to dynamically control the number and frequency of pumps and the supply water temperature based on the fluctuating outdoor dry bulb temperature and building cooling load. The pre-training results indicate that DQN achieved energy-saving rates of 4.70% for the training data and 4.65% for the testing data, while Q-learning performed at -0.66% and 1.28% respectively, indicating that both agents outperformed historical control strategies, thereby demonstrating the effectiveness of the pre-training framework. After pre-training, each agent was deployed back into the real-world system for two days of field validation. During deployment, both agents outperformed the rule-based control, with DQN achieving an energy-saving rate of 9.28% and Q-learning achieving 9.04%, demonstrating that the proposed framework enables RL agents to continue real-world learning with an enhanced control strategy. This study provides a novel pre-training paradigm for implementing RL agents in supply-side control of HVAC systems, potentially enhancing both RL deployment and its online evolution.