Data Cleansing Processing using Pentaho Data Integration: Case Study Data Deduplication

Abstract

Now is the era of data. Every field has data and uses it to progress towards an innovative future. But often, the amount of data that is not balanced with good data quality ranges from differences in data formats, duplicate data, and errors in the data input process. One technique for maintaining and improving data quality is the data cleansing technique. This paper aims to propose data cleansing processing in the case of data deduplication cases using Pentaho Data Integration tools. Pentaho Data Integration done in 4 phases: Analyze, Mapping function, Design and setting, and Evaluation and test. PDI results are tested and compared with the Talend Open Studio tool. The dataset tested was data on factory names at a company in Indonesia tasked with overseeing the distribution of medicines and food. This research is expected to meet the needs of companies, especially in the field of data quality management, especially cases of data duplication and to find out the comparative results of the tools used.