Enhanced Duplicate Count Strategy: Towards New Algorithms to Improve Duplicate Detection

Abstract

Duplicate detection is the process of detecting multiple representations of same real world's entities. Nowadays, data is known to be heterogeneous, and the larger it becomes, the pairwise comparisons number grows highly as well which makes the task more complex. In recent years, many approaches have been developed and attempted to reduce the number of record pair's comparisons in the process while maintaining high matching quality. There are two well-known algorithms, which are the Sorted Neighborhood Method (SNM) and the Blocking algorithms. Being inspired by both algorithms, we propose an Enhanced Duplicate Count Strategy which is a new hybrid approach that creates iterative blocks using windows with dynamic size. It is based on comparing next element with last duplicate found in the current window. Consequently, comparisons are saved, and similarity distance is minimized, which can lead to higher matching quality.