Clustering Web Retrieval Results Accompanied by Removing Duplicate Documents

Abstract

Since keyword-based search engine usually return large amount of results in which there are many unrelated documents and many documents with same content, automatic clustering technology is used to classify the retrieval results. While there are large amount of Web retrieval results, the clustering process usually costs long time and the clusters are not friendly to users since there are still many documents with same content. This paper proposed an improved clustering method by removing the duplicate documents from retrieval results. The removal operation is executed first in initial partition stage during clustering. Then it is executed again after the initial partition stage to remove the duplicate documents thoroughly. We proposed an efficient removal method in this stage. At last, we made experiment to verify our method.