PROWL: Towards Predicting the Runtime of Batch Workloads

Many applications in the enterprise domain require batch processing to perform business critical operations. Batch jobs perform automated, complex processing of large volumes of data without human intervention. Parallel processing allows multiple batch jobs to run concurrently to minimize the total completion time. However, this may result in one or more jobs exceeding their individual completion deadline due to resource sharing. The objective of this work is to predict the completion time of a batch job when it is running in conjunction with other batch jobs. Batch jobs may be multi-threaded and threads can have distinct CPU requirements. Our predictions are based on a simulation model using the service demand (total CPU time required) of each thread in the job. Moreover, for multi-threaded jobs, we simulate the server with instantaneous CPU utilization of each job in the small intervals instead of aggregate value while predicting the completion time. In this paper, a simulation based method is presented to predict the completion time of each batch job in a concurrent run of multiple jobs. A validation study with synthetic benchmark FIO shows that the job completion time prediction error is less than 15% in the worst case.