Coupling time-scale reinforcement learning methods for building operational optimization with waste heat

This paper focuses on the joint optimization of fan coil units (FCUs) and heat pumps in HVAC systems for multi-zone building environments. The core problem involves balancing fast local control of FCUs with slower global control of heat pumps to ensure energy efficiency and indoor comfort. To tackle this, we propose a coupling time-scale reinforcement learning (RL) algorithm, specifically a Deep Q-Network (DQN)-based approach that employs a multi-task learning network to manage both FCU and heat pump control efficiently through the utilization of shared state information. Moreover, the agents are trained and make decisions collaboratively across different timescales. In addition, we develop a high-fidelity building simulation that incorporates detailed thermal models, dynamic loading, and waste heat modules to evaluate the performance of the system in real-world conditions. The experimental results show that the proposed coupling time-scale DQN algorithm improves the accuracy of temperature control by 35.4 % and 26.21 % compared to traditional DQN and the occupant-centric control (OCC) algorithms. Additionally, it reduces regional power fluctuations by 25.18 % and 56.74 % relative to these traditional algorithms. Simultaneously, the proposed algorithm achieves the lowest heat pump energy consumption (2964 W), outperforming traditional DQN (2977 W) and OCC (3051 W) respectively, while maintaining superior temperature control accuracy. These quantitative improvements collectively demonstrate the proposed algorithm’s capability to synergistically balance thermal comfort, power fluctuation, and energy consumption.