A Dynamic Load Balance Algorithm for the S4 Parallel Stream Processing Engine

Abstract

Large streams of data can be analyzed in realtimeby Parallel Stream Processing Engines (PSPEs) which arebased on a graph paradigm where vertices represent processingelements (PEs) and edges represent flows of data among PEs. Inthis work, we propose a new elastic strategy for the S4 PSPE toadjust the overall load of PEs in accordance with the utilizationlevels and data traffic at each PE. Our approach exploits aproducer/consumer model to achieve load balance where newworkers pull events from a buffer queue in order to release theamount of traffic in an overloaded PE. Results show that theproposed strategy prevents saturation of PEs and improves theoverall throughput of the system by up to 470%.