Data partitioning over data streams based on change-aware sampling

Abstract

A novel data partitioning method adapted to a distributed parallel streams processing system for power industry is proposed. This method uses change-aware sampling algorithm that can guarantee low error to describe the distribution characteristics of the data-values first. And then it uses an improved heuristic constructing equal depth histograms algorithm to generate approximate partition vector efficiently. Experiments results on actual data prove that the proposed method is efficient, practical and suitable for time-varying data streams processing