Are large language models qualified reviewers in originality evaluation?

Abstract

Large language models (LLMs) are a new generation of conversational language model with impressive semantic comprehension, text generation, and knowledge inference capabilities. LLMs are significantly influencing the development of science by assisting researchers in analyzing, understanding, and grasping original knowledge in scientific papers. This study investigates LLMs’ potential as qualified reviewers in originality evaluation in zero-shot learning, utilizing a unique, manually crafted prompt. Using biomedical papers as the data source, we constructed two evaluation datasets based on Nobel Prize papers and disruptive index. The evaluation performance of multiple LLMs of different types and scales on the datasets was scrutinized through the analysis of originality score (OS), originality type (OT), and originality description (OD), all of which were generated by the LLM. Our results show that LLMs can to some extent discern papers with distinct originality level via OS; however, they appear to be overly lenient reviewers. In LLMs’ evaluation mechanism, five distinct OTs reflecting varied research contributions do not manifest independently, but together they positively influence OS. Of all the LLMs analyzed, GPT-4 stood out as able to produce the most readable ODs, effectively explaining the inference process for both OS and OT from the perspectives of completeness, logicality, and regularity.