Clustering sentence level-text using fuzzy hierarchical algorithm

Abstract

Clustering is a popular technique for unsupervised text analysis, often used to explore the content of large amounts of sentences. It is performed based on the similarity of sentences. Sentences may contain interrelated concepts and implementing flat clustering algorithms allows one sentence to be present only in one cluster. Also, sentences are semantically related to each other and so word co-occurrence is not a valid measure for sentence level flat clustering. So, WordNet based semantic similarity measure along with fuzzy sentence clustering algorithm is proposed. The existing system uses the Fuzzy C-Means algorithm where the cluster size should be specified as an input. Due to the rigorous convergence criteria, the time complexity is much larger. Most of the NLP documents are hierarchical in nature and so fuzzy hierarchical sentence clustering algorithm is used here. Labeling is performed for each cluster depending on the hierarchy formed and instead of considering all the terms in a sentence, only the verbs and nouns are considered for the similarity computation. Agglomerative clustering based on the verbs and divisive clustering based on nouns is proposed. This methodology is validated through various performance measures like Purity, Entropy and Time. Upon comparing the results for various datasets, it was observed that the overall improvement in purity is 36.6% and entropy is 31%. The time complexity of the hierarchical algorithm is very much less than the EM algorithm. Thus better quality clusters are formed in comparatively less time by using the Fuzzy Hierarchical Sentence Clustering Algorithm.