Exploring the power of resource allocation for Spark executor

Abstract

Nowadays Spark has been widely adopted as a sharp blade in solving big data problems by pipelining tasks of jobs on each node of cluster. In order to improve cluster resource utilization, lots of Spark performance-tuning advices have been proposed both by Spark and researchers. However, we notice that most of these advices focus tuning configuration items in isolation without considering job characteristics. In this paper, we try to explore the impact of executor quota allocation for Spark job in consideration of job stages and size of input. Dozens of carefully designed experiments reveal that execution time among job stages varies in probability as executor quota changes and thus the job execution time varies. We believe this conclusion helps to shed light on allocating executor resource quota regarding to job characteristics.