Optimizing K-means text document clustering using latent semantic indexing and pillar algorithm

Abstract

Document clustering is an important tool to help managing the vast amount of digital text document. This paper introduces a new approach to cluster text document. First, text is preprocessed and indexed using inverted index. Then the index is trimmed using TF-DF thresholding. After that, Term Document Matrix is built based on TF-IDF. Next step uses Latent Semantic Indexing to extract important feature from Term Document Matrix. The following process is selecting seeds via Pillar algorithm. Based on determined seeds, K-Means clustering is performed. Experiment result proves that this approach outperforms standard K-Means document clustering.