Data Reduction with Real-Time Critical Data Forwarding for Internet-of-Things

Abstract

Proliferation of Internet-of-Thing (IoT) has introduced huge amounts of connected devices around the globe. All these connected devices are generating enormous amount of data in frequency of second or in some cases close to millisecond. It is a challenge to tackle the ingest of these "big data". We start to observe bottleneck in term of network bandwidth, storage space as well as computational cost. Therefore, people start putting attention into reducing the size of generated data before it flows to endpoints. We identify some works which work into this direction, however those solutions require certain requirements to be fulfilled, for instance space for caching and certain setup of hardware. This paper presents data reduction algorithm with realtime critical data forwarding. The experiment shows that by only forwarding 31% of data, in best case, we can achieve accuracy 0.97, at same time the algorithm detects critical data and forward to endpoint at real time.