Data Aggregation and Anomaly Detection System for Isomerism and Heterogeneous Data

With the development of big data technology, data accessed by big data platforms maintain the features of mass, isomerism, heterogeneous, and streaming. Therefore, how to access the varied data sources of isomerism and heterogeneous data and how to process and analyze the data become the current challenges. In this paper, we design and implement a data aggregation and anomaly detection system for isomerism and heterogeneous data. The system proposes a novel isomerism and heterogeneous data access sub-system. The sub-system applies improved Avro as the unified data description format and presents different storage algorithms for data serialization to raise the data adaption efficiency. The system adopts Kafka as the message middleware for data aggregation and distribution. Also, we design the anomaly detection and alarming sub-system for detecting the anomalies of streaming data on time and notifying the users. The data aggregation and anomaly detection system has passed all the tests and applied in small and medium-sized enterprises.