Uncertainty-aware Weighted Fair Queueing for Routers Based on Deep Reinforcement Learning

In current computer communication networks, the increasing packet loss and delay caused by the increasing traffic becomes the bottleneck for the desired Quality of Service (QoS). Weighted Fair Queueing can be used to provide differentiated services according to the Service Level Agreement (SLA) associated with each packet. However, due to inaccurate measurements of queue usage, drop rate and delay in real routers, and the intrinsic property of a real network system that there will always be some unpredictable traffic patterns, current methods for WFQ updating can be improved and extended further. In this work, an uncertainty-aware soft actor-critic agent is introduced. First, the learned weights updating strategy is a maximum entropy policy, which is robust under estimation and model error. Second, the technique of model uncertainty estimation is adopted into the agent so that it is capable of detecting novel states that are unseen during the training period, which facilitates a strategy switching framework. The proposed algorithm shows the potential of using reinforcement learning for WFQ weights updating and is compatible with existing techniques by monitoring the model uncertainty, which makes a more robust and stable system. The benefits of applying the proposed algorithm is validated through the simulation studies, showing a promising direction for further exploration.