An Improved Task Scheduling Algorithm Based on Cache Locality and Data Locality in Hadoop

Abstract

The optimization of task scheduling in Hadoop environment is an important research topic. The result of task scheduling affects the system performance and resource utilization. The existing task scheduling algorithm is lack of consideration at the cache level, which makes the performance of the task greatly affected. Therefore, this paper proposes an improved task scheduling algorithm based on cache locality and data locality. Firstly section matrix and weighted bipartite graph are constructed according to the relation between resources and tasks. Then the bipartite graph matching is used to realize map task scheduling for optimizing the local cache and data locality and reducing the data transmission amount during task execution process. The experimental results show that the proposed algorithm can effectively improve the data locality and system performance, which is better than other two algorithms.