A Concept of an In-Memory Database for IoT Sensor Data

Abstract

In the context of digital transformation and use of Industry 4.0 technology in companies, machines and other objects are increasingly being equipped with sensors. Normally, these machines are monitored 24/7, so that data streams are continuously generated by sensors. These data has to be stored in a database. In order to facilitate a fast data mining process and the use of machine learning algorithms, a performant and robust data store for the vast amount of sensor data is necessary. These raw time series sensor data has typical structures that are difficult to model with traditional database management systems. Here, column-oriented In-Memory databases like SAP HANA or Gorilla are better suited. However, SAP HANA have not been developed to store relational data, so that it contains components like transaction and concurrency control, which are unnecessary for the named range of application, because machine learning algorithms only need reading access. By reducing this concept to the essentials, a specialized, lightweight In-Memory database management system can be developed, which perfectly fits to the characteristics of time series sensor data. For that concept the benefits of the In-Memory data structure of SAP HANA and Facebook Gorilla are merged and combined with additional meta information like limits for minimum and maximum warning for each sensor, special user specified column fields or rules for sampling and replenishment values. The evaluation of the implemented prototype shows on the one hand that the time series sensor data can be stored efficiently using a new table structure and an intelligent combination of the ZFP compression method with a block orientated data structure, which results in a good insert performance. On the other hand, this storage logic leads to an efficient data access of the compressed in-memory data structure, thus every reporting or analyzing tasks access the data efficiently and fast.