A Planner-Guided Scheduling Strategy for Multiple Workflow Applications

Abstract

Workflow applications are gaining popularity in recent years because of the prevalence of cluster environments. Many algorithms have been developed since, however most static algorithms are designed in the problem domain of scheduling single workflow applications, thus not applicable to a common cluster environment where multiple workflow applications and other independent jobs compete for resources. Dynamic scheduling approaches can handle the mixed workload practically by nature but their performance has yet to optimize as they do not have a global view of workflow applications. Recent research efforts suggest merging multiple workflows into one workflow before execution, but fail to address an important issue that multiple workflow applications may be submitted at different times by different users. In this paper, we propose a planner-guided dynamic scheduling strategy for multiple workflow applications, leveraging job dependence information and execution time estimation.Our approach schedules individual jobs dynamically without requiring merging the workflow applications a priori. The simulation results show that the proposed algorithm significantly outperforms two other algorithms by 43.6% and 36.7% with respect to workflow makespan and turnaround time respectively, and it performs even better when the number of concurrent workflow applications increases and the resources are scarce.