A deep multi-agent reinforcement learning approach for the micro-service migration problem with affinity in the cloud

Abstract

This paper focuses on the micro-service migration problem with affinity, stemming from the cloud computing industry. Because of periodically creating and deleting micro-services to satisfy users’ demands, the deployment of micro-services in the cloud needs to be regularly adjusted, which is referred to as a micro-service migration. An optimal migration schedule should minimize the number of activated physical machines as well as maximize total internal invoking traffic (affinity). A cooperative multi-agent reinforcement learning (MARL) is proposed, which is enhanced by integrating Hindsight Reward Shaping and by fine-tuning the state encoder using a pre-trained ResNet model. The proposed MARL is validated on both synthetic datasets and real cloud traces of ByteDance and Alibaba, compared with four baseline algorithms: Migration Ant Colony Optimization, Migration Neighborhood Search, Single-Agent Reinforcement Learning, and the optimization solver CPLEX. Finally, an evaluation mechanism called Matching Score is proposed to explain the superior performance of MARL.