Definition of an Architecture for Dynamic and Automatic Checkpoints on Apache Spark

Abstract

Towards a scenario where failures on large-scale systems are inevitable, fault tolerant mechanisms must be efficiently applied. Checkpoint is a widely used technique that consists in saving data states for a fast recovery in case of failure. On Apache Spark – framework that uses in-memory data abstraction –, checkpoint serves to store datasets in a reliable source, so it helps on recovery process of complex datasets. However, once checkpoints must be defined by developer via source code, it may be a hard challenge to choose proper checkpoint scenarios. Therefore, this work proposes an automatic mechanism for checkpoint on Spark, which consists in monitoring system behavior and taking automatic checkpoint process according to defined policies.