HeatPipe: High Throughput, Low Latency Big Data Heatmap with Spark Streaming

Abstract

Heatmap visualization is a well-known type of visualization to alleviate the overplot problem of point visualization. As such, it is well suited to visualize Big Data. In order to tackle the velocity problem of Big Data, one has to leverage streaming computations. Recently, canopy clustering was shown to be well suited for Big Data heatmap visualization. In this article, we present how to design a streaming algorithm to compute canopy clustering using Apache Spark. This result is directly applicable to be included into a lambda architecture.