A Run Time Technique for Handling Error in User-Estimated Execution Times on Systems Processing MapReduce Jobs with Deadlines

Abstract

Effective management of resources on a cloud or cluster is crucial for achieving the quality of service requirements of users, which are typically captured in service level agreements (SLAs). This paper focuses on improving the robustness of resource allocation and scheduling techniques that process an open stream of MapReduce jobs with SLAs, by introducing techniques to handle errors/inaccuracies in user-estimated execution times that are submitted as part of the job's SLA. Inaccuracies in the estimates of task execution times can prevent the resource allocation and scheduling algorithm from making effective scheduling decisions, leading to a degradation in system performance. Techniques for handling error during runtime are presented to handle the situation where jobs have already started executing and their estimated execution times are inaccurate. A simulation-based performance evaluation of the error handling techniques is conducted, which demonstrates that the techniques are effective in improving system performance.