Enhancing fog load balancing through lifelong transfer learning of reinforcement learning agents

Abstract

Fog computing is a promising paradigm for processing Internet of Things (IoT) data. Load balancing (LB) optimizes Fog performance through efficient resource allocation, improving resource utilization, latency for real-time IoT applications, and users’ quality of service. In this work, we enhance the learning process of privacy-aware Reinforcement Learning (PARL), which requires significant training to minimize waiting delays by reducing the number of queued requests without explicitly observing Fog load or resource capabilities. To achieve this, we explore different Transfer Learning (TL) techniques for efficient adaptation to variations in demand, triggering a fine-tuning process when abrupt surges in generation rates are detected. This exploration highlights the advantages and disadvantages of reusing previously learned policies (knowledge) and interactions (experience) over multiple learning epochs with increased difficulties. Our results show that Full TL (using knowledge and experience) enhances the learning and generalization of the PARL agent, allowing it to consistently converge to the optimal solution with 80% less training compared to without TL. Additionally, we propose a lifelong learning framework for practical agent deployment in frequently changing environments. Introducing TL in this framework significantly reduces the computationally expensive training phase compared to training from scratch. Instead of continuous adaptation through ongoing training, balancer resources are preserved to provide faster decisions via a lightweight inference model. In case of significant system changes, the model is swiftly fine-tuned using TL. Furthermore, the framework leverages existing (expert) or simulation-trained agents to initialize newly deployed agents in the network, reducing failure probability in new environments compared to learning from scratch. To our knowledge, no existing efforts in the literature use TL to address lifelong learning for practical RL-based Fog LB. This gap highlights the need for a practical yet efficient solution that minimizes the cost of continuous adaptation to changing conditions.