Multi-Objective Network Congestion Control via Constrained Reinforcement Learning

Traditional congestion control algorithms rely on various model-based methods to improve the end-to-end (E2E) performance of packet transmission. The resulting decisions quickly become less effective amid the dynamics of network conditions. In order to perform congestion control adaptively, reinforcement learning (RL) can be adopted to continuously learn the optimal strategy from the network environment. Oftentimes, the reward of such a learning problem is a weighted sum of multiple E2E performance metrics, such as throughput, delay, and fairness. Unfortunately, those weights can be only manually tuned based on extensive experiments. To address this issue, in this paper, we design a constrained RL algorithm for congestion control named CRL-CC to adaptively tune those weights, with the objective of effectively improving the overall E2E packet transmission performance. In particular, the multi-objective optimization problem is firstly formulated as a constrained optimization problem. Then, the Lagrangian relaxation method is leveraged to transform the constrained optimization problem into a single-objective optimization problem, which is solved by designing a multi-objective reward function with Lagrangian multipliers. Extensive experiments based on OpenAI-Gym show that the proposed CRL-CC algorithm can achieve higher overall performance in various network conditions. In particular, the CRL-CC algorithm outperforms the benchmark algorithm on Pantheon by 21.7%, 27.4%, and 5.3% in throughput, delay, and fairness, respectively.