Synthetic workload generation for load-balancing experiments

Abstract

The Dynamic Workload Generator accurately replays measured workloads in the presence of competing foreground tasks. We have used this workload-generation tool to predict the relative speedups of different sites for an incoming task in our prototype system, using only the resource-utilization patterns observed before the task arrives. Our results show that the load-balancing policies learned by our system effectively exploit idle resources of a distributed computer system.Dynamic Workload Generator is a facility for generating realistic and reproducible synthetic workloads for use in load-balancing experiments. For such experiments, the generated workload must not only mimic the highly dynamic resource-utilization patterns found on today's distributed systems but also behave as a real workload does when test jobs run concurrently with it. The latter requirement is important in testing alternative load-balancing strategies, a process that requires running the same job multiple times, each time at a different site but under an identical network-wide workload.Parts of DWG are implemented inside the operating-system kernel and have complete control over the utilization levels of four key resources: CPU, memory, disk, and network. Besides accurately replaying network-wide load patterns recorded earlier, DWG gives up a fraction of its resources each time a new job arrives and reclaims these resources upon job completion. Pattern-doctoring rules implemented in DWG control the latter operation. This article presents DWG's architecture, its doctoring rules, systematic methods for adjusting and evaluating doctoring rules, and experimental results on a network of Sun workstations.