可以用机器学习来评估已发表学术期刊文章的质量吗?

IF 4.1 Q1 INFORMATION SCIENCE & LIBRARY SCIENCE Quantitative Science Studies Pub Date : 2022-02-22 DOI:10.1162/qss_a_00185
M. Thelwall
{"title":"可以用机器学习来评估已发表学术期刊文章的质量吗?","authors":"M. Thelwall","doi":"10.1162/qss_a_00185","DOIUrl":null,"url":null,"abstract":"Abstract Formal assessments of the quality of the research produced by departments and universities are now conducted by many countries to monitor achievements and allocate performance-related funding. These evaluations are hugely time consuming if conducted by postpublication peer review and are simplistic if based on citations or journal impact factors. I investigate whether machine learning could help reduce the burden of peer review by using citations and metadata to learn how to score articles from a sample assessed by peer review. An experiment is used to underpin the discussion, attempting to predict journal citation thirds, as a proxy for article quality scores, for all Scopus narrow fields from 2014 to 2020. The results show that these proxy quality thirds can be predicted with above baseline accuracy in all 326 narrow fields, with Gradient Boosting Classifier, Random Forest Classifier, or Multinomial Naïve Bayes being the most accurate in nearly all cases. Nevertheless, the results partly leverage journal writing styles and topics, which are unwanted for some practical applications and cause substantial shifts in average scores between countries and between institutions within a country. There may be scope for predicting articles’ scores when the predictions have the highest probability.","PeriodicalId":34021,"journal":{"name":"Quantitative Science Studies","volume":"3 1","pages":"208-226"},"PeriodicalIF":4.1000,"publicationDate":"2022-02-22","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"7","resultStr":"{\"title\":\"Can the quality of published academic journal articles be assessed with machine learning?\",\"authors\":\"M. Thelwall\",\"doi\":\"10.1162/qss_a_00185\",\"DOIUrl\":null,\"url\":null,\"abstract\":\"Abstract Formal assessments of the quality of the research produced by departments and universities are now conducted by many countries to monitor achievements and allocate performance-related funding. These evaluations are hugely time consuming if conducted by postpublication peer review and are simplistic if based on citations or journal impact factors. I investigate whether machine learning could help reduce the burden of peer review by using citations and metadata to learn how to score articles from a sample assessed by peer review. An experiment is used to underpin the discussion, attempting to predict journal citation thirds, as a proxy for article quality scores, for all Scopus narrow fields from 2014 to 2020. The results show that these proxy quality thirds can be predicted with above baseline accuracy in all 326 narrow fields, with Gradient Boosting Classifier, Random Forest Classifier, or Multinomial Naïve Bayes being the most accurate in nearly all cases. Nevertheless, the results partly leverage journal writing styles and topics, which are unwanted for some practical applications and cause substantial shifts in average scores between countries and between institutions within a country. There may be scope for predicting articles’ scores when the predictions have the highest probability.\",\"PeriodicalId\":34021,\"journal\":{\"name\":\"Quantitative Science Studies\",\"volume\":\"3 1\",\"pages\":\"208-226\"},\"PeriodicalIF\":4.1000,\"publicationDate\":\"2022-02-22\",\"publicationTypes\":\"Journal Article\",\"fieldsOfStudy\":null,\"isOpenAccess\":false,\"openAccessPdf\":\"\",\"citationCount\":\"7\",\"resultStr\":null,\"platform\":\"Semanticscholar\",\"paperid\":null,\"PeriodicalName\":\"Quantitative Science Studies\",\"FirstCategoryId\":\"1085\",\"ListUrlMain\":\"https://doi.org/10.1162/qss_a_00185\",\"RegionNum\":0,\"RegionCategory\":null,\"ArticlePicture\":[],\"TitleCN\":null,\"AbstractTextCN\":null,\"PMCID\":null,\"EPubDate\":\"\",\"PubModel\":\"\",\"JCR\":\"Q1\",\"JCRName\":\"INFORMATION SCIENCE & LIBRARY SCIENCE\",\"Score\":null,\"Total\":0}","platform":"Semanticscholar","paperid":null,"PeriodicalName":"Quantitative Science Studies","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/10.1162/qss_a_00185","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"INFORMATION SCIENCE & LIBRARY SCIENCE","Score":null,"Total":0}
引用次数: 7

摘要

摘要许多国家现在对系和大学的研究质量进行正式评估,以监测成果并分配与绩效相关的资金。如果通过发表后的同行评审进行,这些评估将非常耗时,如果基于引文或期刊影响因素,则会过于简单。我研究了机器学习是否可以通过使用引文和元数据来学习如何从同行评审评估的样本中对文章进行评分,从而帮助减轻同行评审的负担。一项实验被用来支持这场讨论,试图预测2014年至2020年所有Scopus窄领域的期刊引文三分之一,作为文章质量分数的代表。结果表明,在所有326个窄域中,可以以高于基线的精度预测这些代理质量三分之一,其中梯度提升分类器、随机森林分类器或多项式朴素贝叶斯在几乎所有情况下都是最准确的。尽管如此,研究结果在一定程度上利用了期刊写作风格和主题,这对于一些实际应用来说是不需要的,并导致国家之间和国内机构之间的平均分数发生了实质性变化。当预测具有最高概率时,可能存在预测文章得分的空间。
本文章由计算机程序翻译,如有差异,请以英文原文为准。
查看原文
分享 分享
微信好友 朋友圈 QQ好友 复制链接
本刊更多论文
Can the quality of published academic journal articles be assessed with machine learning?
Abstract Formal assessments of the quality of the research produced by departments and universities are now conducted by many countries to monitor achievements and allocate performance-related funding. These evaluations are hugely time consuming if conducted by postpublication peer review and are simplistic if based on citations or journal impact factors. I investigate whether machine learning could help reduce the burden of peer review by using citations and metadata to learn how to score articles from a sample assessed by peer review. An experiment is used to underpin the discussion, attempting to predict journal citation thirds, as a proxy for article quality scores, for all Scopus narrow fields from 2014 to 2020. The results show that these proxy quality thirds can be predicted with above baseline accuracy in all 326 narrow fields, with Gradient Boosting Classifier, Random Forest Classifier, or Multinomial Naïve Bayes being the most accurate in nearly all cases. Nevertheless, the results partly leverage journal writing styles and topics, which are unwanted for some practical applications and cause substantial shifts in average scores between countries and between institutions within a country. There may be scope for predicting articles’ scores when the predictions have the highest probability.
求助全文
通过发布文献求助,成功后即可免费获取论文全文。 去求助
来源期刊
Quantitative Science Studies
Quantitative Science Studies INFORMATION SCIENCE & LIBRARY SCIENCE-
CiteScore
12.10
自引率
12.50%
发文量
46
审稿时长
22 weeks
期刊介绍:
期刊最新文献
Technological Impact of Funded Research: A Case Study of Non-Patent References Socio-cultural factors and academic openness of world countries Scope and limitations of library metrics for the assessment of ebook usage: COUNTER R5 and link resolver The rise of responsible metrics as a professional reform movement: A collective action frames account New methodologies for the digital age? How methods (re-)organize research using social media data
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
现在去查看 取消
×
提示
确定
0
微信
客服QQ
Book学术公众号 扫码关注我们
反馈
×
意见反馈
请填写您的意见或建议
请填写您的手机或邮箱
已复制链接
已复制链接
快去分享给好友吧!
我知道了
×
扫码分享
扫码分享
Book学术官方微信
Book学术文献互助
Book学术文献互助群
群 号:481959085
Book学术
文献互助 智能选刊 最新文献 互助须知 联系我们:info@booksci.cn
Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。
Copyright © 2023 Book学术 All rights reserved.
ghs 京公网安备 11010802042870号 京ICP备2023020795号-1