Assessing and predicting the quality of peer reviews: a text mining approach

Abstract

Purpose This paper aims to quantify the quality of peer reviews, evaluate them from different perspectives and develop a model to predict the review quality. In addition, this paper investigates effective features to distinguish the reviews' quality.  Design/methodology/approach First, a fine-grained data set including peer review data, citations and review conformity scores was constructed. Second, metrics were proposed to evaluate the quality of peer reviews from three aspects. Third, five categories of features were proposed in terms of reviews, submissions and responses using natural language processing (NLP) techniques. Finally, different machine learning models were applied to predict the review quality, and feature analysis was performed to understand effective features. Findings The analysis results revealed that reviewers become more conservative and the review quality becomes worse over time in terms of these indicators. Among the three models, random forest model achieves the best performance on all three tasks. Sentiment polarity, review length, response length and readability are important factors that distinguish peer reviews’ quality, which can help meta-reviewers value more worthy reviews when making final decisions. Originality/value This study provides a new perspective for assessing review quality. Another originality of the research lies in the proposal of a novelty task that predict review quality. To address this task, a novel model was proposed which incorporated various of feature sets, thereby deepening the understanding of peer reviews.