SEIZE User Desired Moments: Runtime Inspection for Parallel Dataflow Systems

Abstract

In Data-Intensive Scalable Computing (DISC) Systems, data transformations are concealed by exposed APIs, and intermediate execution moments are masked under dataflow transitions. Consequently, many crucial features and optimizations (e.g., debugging, data provenance, runtime skew detection) are not well-supported. Inspired by our experience in implementing features and optimizations over DISC systems, we present SEIZE, a unified framework that enables dataflow inspection— wiretapping the data-path with listening logic —in MapReduce-style programming model. We generalize our lessons learned by providing a set of primitives defining dataflow inspection, orchestration options for different inspection granularities, and operator decomposition and dataflow puncutation strategy for dataflow intervention. We demonstrate the generality and flexibility of the approach by deploying SEIZE in both Apache Spark and Apache Flink. Our experiments show that, the overhead introduced by the inspection logic is most of the time negligible (less than 5% in Spark and 10% in Flink).