Towards Performance Optimization for Hadoop MapReduce Applications

Apache Hadoop is a widely used open-source distributed platform towards big data processing and provides YARN based distributed parallel processing framework on low cost commodity machines. However, YARN adopts static resource management (that is, the number of containers available per node and the size of each container are static in nature) depending on pre-configured default resource units called containers leading to poor performance to deal with various sort of MapReduce applications. In addition, during the last wave of a job, many available resources occur frequently being idle because YARN does not consider the wave behavior in tasks of MapReduce applications. To take advantage of idle resources resulting in performance improvement, the important parameter, the number of map tasks is needed to optimize based on the available resources and governed by split size. Therefore, this parameter is optimized through the split size tuning based on the available resources. To address the drawback of static resource management of yarn in Hadoop, the numbers of concurrent containers per machine are tuned to optimize the node performance for running each MapReduce application. As per experimental results, the proposed system that optimizes the selected parameter on optimized concurrent containers can achieve the performance gains of MapReduce applications while reducing the optimization overheads.