O-DQR: A Multi-Agent Deep Reinforcement Learning for Multihop Routing in Overlay Networks

Abstract

This paper addresses the problem of dynamic packet routing in overlay networks using a fully decentralized Multi-Agent Deep Reinforcement Learning (MA-DRL). Overlay networks are built by having a virtual topology on top of an Internet Service Provider (ISP) underlay network, where those nodes are running a fixed, single path routing policy decided by the ISP. In such a scenario, the underlay topology and the traffic are unknown by the overlay network. In this setting, we propose O-DQR, which is an MA-DRL framework working under Distributed Training Decentralized Execution (DTDE), where the agents are allowed to communicate only with their immediate overlay neighbors during both training and inference. We address three fundamental aspects for deploying such a solution: (i) performance (delay, loss rate), where the framework can achieve near-optimal performance, (ii) control overhead, which is reduced by enabling the agents to send control packets only when needed dynamically; and (iii) training convergence stability, which is improved by proposing a guided reward mechanism for dynamically learning the penalty applied when a packet is lost. Finally, we evaluate our solution through extensive experimentation in a realistic network simulation in both offline training and continual learning settings.