Towards a Dataflow Runtime Environment for Edge, Fog and In-Situ Computing

Abstract

In the dataflow computation model, instructions or tasks are fired according to their data dependencies, instead of following program order, thus allowing natural parallelism exploitation. Dataflow has been used, in different flavors and abstraction levels (from processors to runtime libraries), as an interesting alternative for harnessing the potential of modern computing systems. Sucuri is a dataflow library for Python that allows users to specify their application as a dependency graph and execute it transparently at clusters of multicores, while taking care of scheduling issues. Recent trends in Fog and In-situ computing assumes that storage and network devices will be equipped with processing elements that usually have lower power consumption and performance. An important decision on such system is whether to move data to traditional processors (paying the communication costs), or performing computation where data is sitting, using a potentially slower processor. Hence, runtime environments that deal with that trade-off are extremely necessary. This work takes a first step towards a solution that considers Edge/Fog/In-situ in a dataflow runtime. We use Sucuri to manage the execution in a small system with a regular PC and a Parallella board. Experiments with text processing applications running with different input sizes, network latency and packet loss rates allow a discussion of scenarios where this approach would be fruitful.