Sample-Efficient Multi-Agent DQNs for Scalable Multi-Domain 5G+ Inter-Slice Orchestration

Abstract

Data-driven network slicing has been recently explored as a major driver for beyond 5G networks. Nevertheless, we are still a long way before such solutions are practically applicable in real problems. Most solutions addressing the problem of dynamically placing virtual network function chains (“slices”) on top of a physical topology still face one or more of the following hurdles: (i) they focus on simple slicing setups (e.g. single domain, single slice, simple VNF chains and performance metrics); (ii) solutions based on modern reinforcement learning theory have to deal with astronomically high action spaces, when considering multi-VNF, multi-domain, multi-slice problems; (iii) the training of the algorithms is not particularly data-efficient, which can hinder their practical application given the scarce(r) availability of cellular network related data (as opposed to standard machine learning problems). To this end, we attempt to tackle all the above shortcomings in one common framework. For (i), we propose a generic, queuing network based model that captures the inter-slice orchestration setting, supporting complex VNF chain topologies and end-to-end performance metrics. For (ii), we explore multi-agent DQN algorithms that can reduce action space complexity by orders of magnitude compared to standard DQN. For (iii), we investigate two mechanisms to store to and select from the experience replay buffer, in order to speed up the training of DQN agents. The proposed scheme was validated to outperform both vanilla DQN (by orders of magnitude faster convergence) and static heuristics ( $3\times $ cost improvement).