Fault-Tolerant Orchestration of Bags-of-Tasks with Application-Directed Checkpointing in a Distributed Environment

Abstract

A wide spectrum of applications, ranging from big data analytics to financial risk modeling and genomics, feature a high degree of parallelism, forming bags-of-tasks. Such applications are typically processed on distributed resources and are often prone to transient software failures. Consequently, load balancing and fault tolerance are two crucial aspects of such environments. In this paper, we consider bag-of-tasks jobs that utilize application-directed checkpointing. According to this technique, each component task is responsible for checkpointing its own progress at regular intervals during its execution. When a failure occurs, the affected task is rolled back to its most recent checkpoint and resumes execution. In order to investigate the impact of transient software failures on the performance of the system, we employ four resource allocation strategies, two well-known and two novel ones. The routing policies are compared through simulation experiments, under different task failure probabilities and load cases.