Scheduling divisible reduce tasks in MapReduce

Abstract

The computations in MapReduce are composed of map and reduce tasks. Although performance of map tasks has been investigated extensively, most researches ignore the scheduling of reduce tasks. This paper proposes a divisible load scheduling model for reduce tasks in a MapReduce job. By analyzing intermediate data transmission and reduce task execution in reduce phase, reduce tasks are abstracted as divisible loads. The optimal scheduling of reduce tasks is solved with linear programming. The performance is evaluated under different environments. Experiment results show that at least 40% performance improvement is achieved with the optimal scheduling.