Rank-IDF: A Statistical and Network Based Feature Words Selection in Big Data Text Analysis

Abstract

As big data for text has been one of the core data types in the era of artificial intelligence, feature words selection technique become increasingly important in big data text analysis. The traditional statistical TF-IDF feature words selection algorithm lacks the semantic information extraction ability of text, while the network model Textrank applies the sentence semantic features to feature calculation between words. Network model such as Textrank is very suitable for text feature selection, but it does not take influencing factors of the relationship between documents into consideration, so common words appearing frequently in feature words selected result. Based on the analysis of both feature words selection method, this paper raises a combination of statistical and network model integrated the advantages of Textrank and TF-IDF, and proposes a text feature selection method based on Rank-IDF. The Rank-IDF algorithm has better feature selection and common word filtering effects.