Dragon: A Lightweight, High Performance Distributed Stream Processing Engine

Abstract

The performance of a distributed stream processing engine is traditionally considered in terms of fundamental measurements of latency and throughput. Recently, Apache Storm has demonstrated sub-millisecond latencies for inter-component tuple transmission, though it does so through aggressive throttling that leads to strict throughput limitations in order to keep tuple queues near empty. On the other hand, Apache Heron has excellent throughput characteristics, especially when operating near unstable conditions, but its inter-component latencies typically start around 10 milliseconds. Both of these systems require roughly 650MB of installation space. We have developed Dragon, loosely based on the same API as Storm and Heron, that is both lightweight, requiring just 7.5MB of installation space, and competitive in performance to Storm and Heron. In this paper we show experiments with all three systems using the Word Count benchmark. Dragon achieves throughput characteristics near to that of Heron and inter-component latencies less than 10ms under high load. In particular, Dragon’s maximum latency is significantly less that Storm’s maximum latency under high load. Finally Dragon managed to remain stable at higher effective throughput than Heron. We believe Dragon is a good "allrounder" solution and is particularly suitable for Edge computing applications, given its small installation footprint.