Portable and Composable Flow Graphs for In Situ Analytics

Abstract

Existing data analysis and visualization algorithms are used in a wide range of simulations that strive to support an increasing number of runtime systems. The BabelFlow framework has been designed to address this situation by providing users with a simple interface to implement analysis algorithms as dataflow graphs portable across different runtimes. The limitation in BabelFlow, however, is that the graphs are not easily reusable. Plugging them into existing in situ workflows and constructing more complex graphs is difficult. In this paper, we introduce LegoFlow, an extension to BabelFlow that addresses these challenges. Specifically, we integrate LegoFlow into Ascent, a flyweight framework for large scale in situ analytics, and provide a graph composability mechanism. This mechanism is an intuitive approach to link an arbitrary number of graphs together to create more complex patterns, as well as avoid costly reimple-mentations for minor modifications. Without sacrificing portability, LegoFlow introduces complete flexibility that maximizes the productivity of in situ analytics workflows. Furthermore, we demonstrate a complete LULESH simulation with LegoFlow-based in situ visualization running on top of Charm++. It is a novel approach for in situ analytics, whereby the asynchronous tasking runtime allows routines for computation and analysis to overlap. Finally, we evaluate a number of LegoFlow-based filters and extracts in Ascent, as well as the scaling behavior of a LegoFlow graph for Radix-k based image compositing.