Data Preparation for Data Mining in Chemical Plants using Big Data

Abstract

Data preparation for data mining in industrial applications is a key success factor which requires considerable repeated efforts. Although the required activities need to be repeated in very similar fashion across many projects, details of their implementation differ and require both application understanding and experience. As a result, data preparation is done by data mining experts with a strong domain background and a good understanding of the characteristics of the data to be analyzed. Experts with these profiles usually have an engineering background and no strong expertise in distributed programming or big data technology. Unfortunately, the amount of data can be so large that distributed algorithms are required to allow for inspection of results and iteration of preparation steps. This contribution introduces an interactive data preparation workflow for signal data from chemical plants enabling domain experts without background in distributed computing and extensive programming experience to leverage the power of big data technologies.