Hybrid Job Scheduling in Distributed Systems based on Clone Detection

Abstract

In order to propose an efficient scheduling policy in a large distributed heterogeneous environment, resource requirements of newly submitted jobs should be predicted prior to the execution of jobs. An execution history can be maintained to store the execution profile of all jobs executed earlier on a given set of resources. The execution history stores the actual CPU cycle consumed by the job as well as the resource details where it is executed. A feedback-guided job-modeling scheme can be used to detect similarity between the newly submitted jobs and previously executed jobs. It can also be used to predict resource requirements based on this similarity. However, efficient resource scheduling based on this knowledge has not been dealt with. In this paper, we propose a hybrid, scheduling policy of new jobs, which are independent of each other, based on their similarity with history jobs. Here we focus on exact clone jobs only i.e. its identical job is found in execution history and predicted resource consumption is same as exact resource consumption. We also endeavor to deal with two conflicting parameters i.e., execution cost and make span of jobs. A comparison with other existing algorithms is also presented in this paper.