Parallel Implementation of Local Similarity Search for Unstructured Text Using Prefix Filtering

Abstract

Identifying partially duplicated text segments among documents is an important research problem with applications in plagiarism detection and near-duplicate web page detection. We investigate the problem of local similarity search for finding partially replicated text, focusing on its parallel implementation. Our aim is to find text windows that are approximately similar in two documents, using a filter verification framework. We present various parallel approaches to the problem, of which input data partitioning along with the reduction of individual index maps was found to be most suitable. We analyzed the effect of varying similarity threshold and number of processes on speedup, and also performed cost analysis. Experimental results show that the proposed method achieves up to 13x speedup on a 24-core processor.