Semantic-based intelligent data clean framework for big data

Abstract

In order to overcome the limitation of existing data cleansing methods working on massive data, in this paper, we propose a generic semantic-based framework using parallelized processing model for effective big data cleansing. We also use an improved Semantic-Based Keyword Matching Algorithm to deal with duplicate data. Experimental results show that this parallelized framework with improved Semantic-Based Keyword Matching Algorithm can identify duplicates with high recall and precision and have a good performance for big data cleansing.