A Compression Algorithm for Fluctuant Data in Smart Grid Database Systems

Abstract

In this paper, we present a lossless compression algorithm for fluctuant data, which can be integrated into database system and allows regular database insertion and queries. The algorithm is based on the observation that fluctuant data, although varied violently during small time intervals, have similar patterns over time. The algorithm first partitioned consecutive k records into segments. Those segments are normalized and treated as vectors in k-dimensional space. Classification algorithms are then applied to find representative vectors for those normalized vectors. The classification criterion is that any segments after normalization can find at least one representative vector such that their distance is less than a given threshold. Those representative vectors, called codes, are stored in a codebook. The codebook can be generated offline from a small training dataset, and used repeatedly. The online compression algorithm searches the nearest code for an input segment, and stores only the ID of the code and their difference. Since the difference is small, it can be compressed by Rice coding or Golomb coding.lossless compression algorithm.