DCSPARK: Virtualizing spark using Docker containers

Abstract

As MapReduce has become a popular model for large-scale data procession in recent years, companies and researchers take advantage of this model to solve their problems. The applications may run on the same MapReduce cluster, with their own system-wide configure settings and library dependencies, respectively. Sometimes, their configure settings and library dependencies are conflicted with each other. How to ensure these applications to run together correctly without mutual interference and achieve high resources utilization gives a challenge to the researchers. In this paper, we propose DCSpark, a framework that leverages the power of Docker containers that allows users to run Spark applications which have conflicting configurations and library dependencies in one physical cluster. In addition, it's presented an implementation of our framework called DCM which is aimed at managing the physical cluster, processing scheduling problem and building the container-based Spark cluster images automatically according to the dependence environment of the applications. Our experimental evaluation shows that DCSpark introduces negligible overhead for CPU and memory performance compared with the native Spark cluster.