Dynamic Models for the Formal Verification of Big Data Applications Via Stochastic Model Checking

Abstract

Big Data Applications (BDAs) manage so much data to require a cluster of machines for computation and storage. Their execution often has temporal constraints, such as deadlines to process the data. BDAs are executed within Big Data Frameworks (BDFs), that provide mechanisms to automatically manage the complexity of the computation distribution. For a BDA to fulfill its deadline when executed in a BDF, online dynamic resource allocation policies should be in place. The introduction of control for such resource allocation calls for formal verification of the closed-loop system. Model checkers verify the correct behaviour of programs, and in principle they could be used to prove properties on the BDF execution. However, the complexity of BDFs makes it infeasible to directly model the BDAs and BDFs. We propose a formalism to associate the execution of a BDA with afirst-principle dynamic simulation model that can be used for model checking in the place of the real application, making the verification viable in practice. We introduce our formalism, apply it to a well assessed framework, and test its capabilities. We show that our solution is able to capture the dynamics and prove properties of the BDA execution using a stochastic model checker.