Deciding Backup Location Methods for Distributed Stream Processing System

Abstract

A large amount of stream data are generated from some devices such as sensors and cameras. These stream data should be timely processed for real-time applications to satisfy the data latency requirements. To process a large amount of data in a short time, utilizing stream processing on edge/fog computing is a promising technology. In the stream processing system, a snapshot of processes and replications of the stream data are stored on another server, and when server fault or load spike of server occurs, the process is continued by using the stored snapshots and replicated data. Therefore, with edge computing environment, which has low bandwidth resource, process recovery takes a long time due to the transferring of restored data. In this paper, we propose a stream processing system architecture to decide servers to store snapshots and replication data and redeploy processes by considering the load of each server and the network bandwidth. We also propose a semi-optimal algorithm that reduces the computational cost by appropriately sorting servers and tasks according to the network bandwidth and server load. The algorithm can find a solution over 1000 times faster than the Coin or Branch and Cut (CBC) solver.