Digital Twin-Accelerated Online Deep Reinforcement Learning for Admission Control in Sliced Communication Networks

Abstract

The proliferation of diverse wireless services has led to emerging technologies for network slicing. Admission control plays a crucial role in achieving service-oriented goals in sliced communication networks through selective acceptance of service requests. To enhance the performance of admission control in intricate contemporary communication networks, deep reinforcement learning (DRL) has been widely adopted to enhance the effectiveness and flexibility of intelligent admission control. However, due to the simulation-to-reality gap, DRL models trained in a simulation environment can face considerable performance degradation when transferred to the deployment environment. Although online DRL tries to avoid such gaps, its expensive trial-and-error cost poses economic and safety concerns for network operators. We propose a cooperative framework integrating digital twin (DT) and online DRL to address this issue. Specifically, a behavior cloning-based DT is established to parameterize a default admission control policy in real networks, and a DT-accelerated online DRL strategy is then developed for further policy optimization. The DT is constructed as a neural network, featuring a customized output layer to address extensive action spaces in queuing systems. Extensive simulations show that the proposed DRL solution facilitates the stability of the online DRL and accelerates the convergence, yielding a resource utilization improvement of up to 26.39% compared to the state-of-the-art DRL model, while maintaining consistent performance with the online DRL method in terms of long-term revenues. Meanwhile, the proposed solution is versatile and adaptable to various DRL-based network optimization tasks.