Randomized load balancing with a helper

In a cloud environment, a scheduler assigns arriving tasks to one of many servers, with the goal of minimizing response times. There are two conventional approaches to cloud scheduling. The first is called the Join-the-Shortest-Queue (JSQ) algorithm, which directs an arriving task to the least loaded server. Despite its excellent delay performance, JSQ is throughput-limited, and thus doesn't scale well with the number of servers. The second is called the Power-of-d-choices (Pod) algorithm, which selects d servers at random and routes a task to the least loaded server of the d servers. Despite its scalability, Pod suffers from long tail response times. In this paper, a hybrid scheduling strategy is proposed, and it consists of a Pod scheduler and a throughput-limited helper. Hybrid scheduling takes the best of both worlds, enjoying scalability and low tail response times. In particular, hybrid scheduling has bounded maximum queue size in the large-system regime, which is in sharp contrast to the Pod scheduling whose maximum queue size is unbounded.