Exploiting statistical and semantic information for document clustering: An evaluation on feature selection

Abstract

Feature selection is not only a key to handle the high dimensionality phenomenon caused by the vector space model representation, but mainly an efficient technique to reduce the noise generated by the irrelevant and redundant terms. However, in order to effectively capture the most important features, both the semantic and the statistical information within the feature space should be taken into account. Thereby, we propose a sequential and a hybrid clustering and feature selection approaches that combines statistical and semantic feature weight estimation in order to select the most informative features. We first perform a comparative study on powerful statistical feature selection methods and an analysis was done for the semantic methods. Then, we extract the best combination of statistical and semantic methods for the sequential and hybrid approaches. Detailed experimental results on three different data sets are provided in this paper.