A predominant statistical approach to identify semantic similarity of textual documents

Abstract

Semantic similarity is the processes of identifying similar words. It relates to computing the similarity between documents which are not lexicographically similar. This paper proposed an empirical method to estimate the semantic similarity using HBase. Specifically this paper defines various word co-occurrence in the document measured and its synonyms are also identified using WordNet. By using the statistical approaches such as MSE and MSD, similarity has been measured. This research focuses on evaluating the similarity between the key document and source documents in the document corpus. In this paper, the developed predominant tool using statistical approach has been tested by checking the similarity of the assignments submitted by the students to check the integrity of a student. This tool may also be used to identify Plagiarism of documents and to eliminate duplicates in a text repository.