Dynamic co-scheduling of distributed computation and replication

Abstract

We are interested in developing the infrastructural tools that allow a distributed data intensive computing environment to be shared by a group of collaborating but geographically separated researchers in an interactive manner, as opposed to a batch mode of operation. However, without advanced reservation, it is difficult to assure a certain level of performance on a large number of shared and heterogeneous servers. To achieve scalable parallel speedups in this scenario, we must closely integrate the management of computation and runtime data movement. In this paper, we first define the canonical scheduling problem for datasets distributed with k-way replication in the wide area. We then develop a dynamic co-scheduling algorithm that integrates the scheduling of computation and data movement. Using time-varying visualization as the driving application, we demonstrate that our co-scheduling approach improves not only application performance but also server utilization at a very reasonable cost.