Leveraging probabilistic segmentation to document clustering

Abstract

In this paper a novel approach to document clustering has been introduced by defining a representative-based document similarity model that performs probabilistic segmentation of documents into chunks. The frequently occuring chunks that are considered as representatives of the document set, may represent phrases or stem of true words. The representative based document similarity model, containing a term-document matrix with respect to the representatives, is a compact representation of the vector space model that improves quality of document clustering over traditional methods.