A productivity improvement of distributed software testing using checkpoint

Abstract

The advancement of storage technologies and the fast-growing number of generated data have made the world moved into the Big Data era. In this past, we had many data mining tools but they are inadequate to process Data-Intensive Scalable Computing workloads. The Apache Spark framework is a popular tool designed for Big Data processing. It leverages in-memory processing techniques that make Spark up to 100 times faster than Hadoop. Testing this kind of Big Data program is time consuming. Unfortunately, developers lack a proper testing framework, which cloud help assure quality of their data-intensive processing programs, while saving development time. We propose Distributed Test Checkpointing (DTC) for Apache Spark, DTC applies unit testing to the Big Data software development life cycle and reduce time spent for each testing loop with checkpoint. From the experimental results, we found that in the subsequence rounds of unit testing, DTC dramatically speed the testing time up to 450–500% faster. In case of storage, DTC can cut unnecessary data off and make the storage 19.7 times saver than the original checkpoint of Spark.