首页 > 最新文献

Research integrity and peer review最新文献

英文 中文
A guide for social science journal editors on easing into open science. 社会科学期刊编辑进入开放科学的指南。
IF 7.2 Q1 ETHICS Pub Date : 2024-02-16 DOI: 10.1186/s41073-023-00141-5
Priya Silverstein, Colin Elman, Amanda Montoya, Barbara McGillivray, Charlotte R Pennington, Chase H Harrison, Crystal N Steltenpohl, Jan Philipp Röer, Katherine S Corker, Lisa M Charron, Mahmoud Elsherif, Mario Malicki, Rachel Hayes-Harb, Sandra Grinschgl, Tess Neal, Thomas Rhys Evans, Veli-Matti Karhulahti, William L D Krenzer, Anabel Belaus, David Moreau, Debora I Burin, Elizabeth Chin, Esther Plomp, Evan Mayo-Wilson, Jared Lyle, Jonathan M Adler, Julia G Bottesini, Katherine M Lawson, Kathleen Schmidt, Kyrani Reneau, Lars Vilhuber, Ludo Waltman, Morton Ann Gernsbacher, Paul E Plonski, Sakshi Ghai, Sean Grant, Thu-Mai Christian, William Ngiam, Moin Syed

Journal editors have a large amount of power to advance open science in their respective fields by incentivising and mandating open policies and practices at their journals. The Data PASS Journal Editors Discussion Interface (JEDI, an online community for social science journal editors: www.dpjedi.org ) has collated several resources on embedding open science in journal editing ( www.dpjedi.org/resources ). However, it can be overwhelming as an editor new to open science practices to know where to start. For this reason, we created a guide for journal editors on how to get started with open science. The guide outlines steps that editors can take to implement open policies and practices within their journal, and goes through the what, why, how, and worries of each policy and practice. This manuscript introduces and summarizes the guide (full guide: https://doi.org/10.31219/osf.io/hstcx ).

期刊编辑拥有很大的权力,可以通过激励和规定期刊的开放政策和实践,在各自领域推动开放科学的发展。Data PASS期刊编辑讨论界面(JEDI,社会科学期刊编辑的在线社区:www.dpjedi.org )整理了几种将开放科学纳入期刊编辑工作的资源( www.dpjedi.org/resources )。然而,作为一名刚刚接触开放科学实践的编辑,可能会不知从何入手。为此,我们为期刊编辑创建了一份指南,指导他们如何开始使用开放科学。该指南概述了编辑在其期刊中实施开放政策和实践的步骤,并介绍了每项政策和实践的内容、原因、方法和担忧。本手稿对指南进行了介绍和总结(指南全文:https://doi.org/10.31219/osf.io/hstcx )。
{"title":"A guide for social science journal editors on easing into open science.","authors":"Priya Silverstein, Colin Elman, Amanda Montoya, Barbara McGillivray, Charlotte R Pennington, Chase H Harrison, Crystal N Steltenpohl, Jan Philipp Röer, Katherine S Corker, Lisa M Charron, Mahmoud Elsherif, Mario Malicki, Rachel Hayes-Harb, Sandra Grinschgl, Tess Neal, Thomas Rhys Evans, Veli-Matti Karhulahti, William L D Krenzer, Anabel Belaus, David Moreau, Debora I Burin, Elizabeth Chin, Esther Plomp, Evan Mayo-Wilson, Jared Lyle, Jonathan M Adler, Julia G Bottesini, Katherine M Lawson, Kathleen Schmidt, Kyrani Reneau, Lars Vilhuber, Ludo Waltman, Morton Ann Gernsbacher, Paul E Plonski, Sakshi Ghai, Sean Grant, Thu-Mai Christian, William Ngiam, Moin Syed","doi":"10.1186/s41073-023-00141-5","DOIUrl":"10.1186/s41073-023-00141-5","url":null,"abstract":"<p><p>Journal editors have a large amount of power to advance open science in their respective fields by incentivising and mandating open policies and practices at their journals. The Data PASS Journal Editors Discussion Interface (JEDI, an online community for social science journal editors: www.dpjedi.org ) has collated several resources on embedding open science in journal editing ( www.dpjedi.org/resources ). However, it can be overwhelming as an editor new to open science practices to know where to start. For this reason, we created a guide for journal editors on how to get started with open science. The guide outlines steps that editors can take to implement open policies and practices within their journal, and goes through the what, why, how, and worries of each policy and practice. This manuscript introduces and summarizes the guide (full guide: https://doi.org/10.31219/osf.io/hstcx ).</p>","PeriodicalId":74682,"journal":{"name":"Research integrity and peer review","volume":"9 1","pages":"2"},"PeriodicalIF":7.2,"publicationDate":"2024-02-16","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10870631/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"139742810","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 0
Librarians and information specialists as methodological peer-reviewers: a case-study of the International Journal of Health Governance. 图书馆员和信息专家作为方法同行评审员:《国际卫生治理杂志》案例研究。
Q1 ETHICS Pub Date : 2024-01-19 DOI: 10.1186/s41073-023-00142-4
Irina Ibragimova, Helen Fulbright

Background: Objectives of this study were to analyze the impact of including librarians and information specialist as methodological peer-reviewers. We sought to determine if and how librarians' comments differed from subject peer-reviewers'; whether there were differences in the implementation of their recommendations; how this impacted editorial decision-making; and the perceived utility of librarian peer-review by librarians and authors.

Methods: We used a mixed method approach, conducting a qualitative analysis of reviewer reports, author replies and editors' decisions of submissions to the International Journal of Health Governance. Our content analysis categorized 16 thematic areas, so that methodological and subject peer-reviewers' comments, decisions and rejection rates could be compared. Categories were based on the standard areas covered in peer-review (e.g., title, originality, etc.) as well as additional in-depth categories relating to the methodology (e.g., search strategy, reporting guidelines, etc.). We developed and used criteria to judge reviewers' perspectives and code their comments. We conducted two online multiple-choice surveys which were qualitatively analyzed: one of methodological peer-reviewers' perceptions of peer-reviewing, the other of published authors' views on the suggested revisions.

Results: Methodological peer-reviewers assessed 13 literature reviews submitted between September 2020 and March 2023. 55 reviewer reports were collected: 25 from methodological peer-reviewers, 30 from subject peer-reviewers (mean: 4.2 reviews per manuscript). Methodological peer-reviewers made more comments on methodologies, with authors more likely to implement their changes (52 of 65 changes, vs. 51 of 82 by subject peer-reviewers); they were also more likely to reject submissions (seven vs. four times, respectively). Where there were differences in recommendations to editors, journal editors were more likely to follow methodological peer-reviewers (nine vs. three times, respectively). The survey of published authors (87.5% response rate) revealed four of seven found comments on methodologies helpful. Librarians' survey responses (66.5% response rate) revealed those who conducted peer-reviews felt they improved quality of publications.

Conclusions: Librarians can enhance evidence synthesis publications by ensuring methodologies have been conducted and reported appropriately. Their recommendations helped authors revise submissions and facilitated editorial decision-making. Further research could determine if sharing reviews with subject peer-reviewers and journal editors could benefit them in better understanding of evidence synthesis methodologies.

背景:本研究旨在分析图书馆员和信息专家作为方法同行评审员的影响。我们试图确定图书馆员的意见与学科同行评审员的意见是否不同以及如何不同;在落实他们的建议方面是否存在差异;这对编辑决策有何影响;以及图书馆员和作者对图书馆员同行评审效用的看法:我们采用了混合方法,对《国际卫生治理期刊》的审稿人报告、作者回复和编辑决定进行了定性分析。我们的内容分析对 16 个主题领域进行了分类,以便对方法和主题同行评审员的评论、决定和退稿率进行比较。这些类别基于同行评审的标准领域(如标题、原创性等)以及与方法论相关的其他深入类别(如检索策略、报告指南等)。我们制定并使用标准来判断审稿人的观点并对他们的评论进行编码。我们进行了两项在线选择题调查,并对其进行了定性分析:一项是方法学同行评审员对同行评审的看法,另一项是发表论文的作者对建议修改的看法:方法学同行评审员评估了2020年9月至2023年3月期间提交的13篇文献综述。共收集到 55 份审稿人报告:25 份来自方法论同行评审员,30 份来自主题同行评审员(平均:每份手稿 4.2 次评审)。方法学同行评审者对方法学提出了更多意见,作者更有可能执行他们的修改意见(65 次修改中的 52 次,而主题同行评审者 82 次修改中的 51 次);他们也更有可能拒绝投稿(分别为 7 次和 4 次)。在向编辑提出建议方面,期刊编辑更倾向于听从方法同行评审员的建议(分别为 9 次和 3 次)。对已发表论文的作者进行的调查(87.5% 的回复率)显示,7 位作者中有 4 位认为方法学方面的意见很有帮助。对图书管理员的调查(66.5% 的回复率)显示,进行同行评审的图书管理员认为同行评审提高了出版物的质量:图书馆员可以通过确保方法的实施和报告的适当性来提高证据综述出版物的质量。他们的建议有助于作者修改提交的论文并促进编辑决策。进一步的研究可以确定,与主题同行评审员和期刊编辑分享评论是否有利于他们更好地理解证据综合方法。
{"title":"Librarians and information specialists as methodological peer-reviewers: a case-study of the International Journal of Health Governance.","authors":"Irina Ibragimova, Helen Fulbright","doi":"10.1186/s41073-023-00142-4","DOIUrl":"10.1186/s41073-023-00142-4","url":null,"abstract":"<p><strong>Background: </strong>Objectives of this study were to analyze the impact of including librarians and information specialist as methodological peer-reviewers. We sought to determine if and how librarians' comments differed from subject peer-reviewers'; whether there were differences in the implementation of their recommendations; how this impacted editorial decision-making; and the perceived utility of librarian peer-review by librarians and authors.</p><p><strong>Methods: </strong>We used a mixed method approach, conducting a qualitative analysis of reviewer reports, author replies and editors' decisions of submissions to the International Journal of Health Governance. Our content analysis categorized 16 thematic areas, so that methodological and subject peer-reviewers' comments, decisions and rejection rates could be compared. Categories were based on the standard areas covered in peer-review (e.g., title, originality, etc.) as well as additional in-depth categories relating to the methodology (e.g., search strategy, reporting guidelines, etc.). We developed and used criteria to judge reviewers' perspectives and code their comments. We conducted two online multiple-choice surveys which were qualitatively analyzed: one of methodological peer-reviewers' perceptions of peer-reviewing, the other of published authors' views on the suggested revisions.</p><p><strong>Results: </strong>Methodological peer-reviewers assessed 13 literature reviews submitted between September 2020 and March 2023. 55 reviewer reports were collected: 25 from methodological peer-reviewers, 30 from subject peer-reviewers (mean: 4.2 reviews per manuscript). Methodological peer-reviewers made more comments on methodologies, with authors more likely to implement their changes (52 of 65 changes, vs. 51 of 82 by subject peer-reviewers); they were also more likely to reject submissions (seven vs. four times, respectively). Where there were differences in recommendations to editors, journal editors were more likely to follow methodological peer-reviewers (nine vs. three times, respectively). The survey of published authors (87.5% response rate) revealed four of seven found comments on methodologies helpful. Librarians' survey responses (66.5% response rate) revealed those who conducted peer-reviews felt they improved quality of publications.</p><p><strong>Conclusions: </strong>Librarians can enhance evidence synthesis publications by ensuring methodologies have been conducted and reported appropriately. Their recommendations helped authors revise submissions and facilitated editorial decision-making. Further research could determine if sharing reviews with subject peer-reviewers and journal editors could benefit them in better understanding of evidence synthesis methodologies.</p>","PeriodicalId":74682,"journal":{"name":"Research integrity and peer review","volume":"9 1","pages":"1"},"PeriodicalIF":0.0,"publicationDate":"2024-01-19","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10797710/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"139491777","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 0
The quizzical failure of a nudge on academic integrity education: a randomized controlled trial. 推动学术诚信教育的奇怪失败:一项随机对照试验。
Q1 ETHICS Pub Date : 2023-11-30 DOI: 10.1186/s41073-023-00139-z
Aurélien Allard, Anna Catharina Vieira Armond, Mads Paludan Goddiksen, Mikkel Willum Johansen, Hillar Loor, Céline Schöpfer, Orsolya Varga, Christine Clavien

Background: Studies on academic integrity reveal high rates of plagiarism and cheating among students. We have developed an online teaching tool, Integrity Games ( https://integgame.eu/ ), that uses serious games to teach academic integrity. In this paper, we test the impact of a soft intervention - a short quiz - that was added to the Integrity Games website to increase users' interest in learning about integrity. Based on general principles of behavioral science, our quiz highlighted the intricacy of integrity issues, generated social comparisons, and produced personalized advice. We expected that these interventions would create a need for knowledge and encourage participants to spend more time on the website.

Methods: In a randomized controlled trial involving N = 405 students from Switzerland and France, half of the users had to take a short quiz before playing the serious games, while the other half could directly play the games. We measured how much time they spent playing the games, and, in a post-experimental survey, we measured their desire to learn about integrity issues and their understanding of integrity issues.

Results: Contrary to our expectations, the quiz had a negative impact on time spent playing the serious games. Moreover, the quiz did not increase participants' desire to learn about integrity issues or their overall understanding of the topic.

Conclusions: Our quiz did not have any measurable impact on curiosity or understanding of integrity issues, and may have had a negative impact on time spent on the Integrity games website. Our results highlight the difficulty of implementing behavioral insights in a real-world setting.

Trial registration: The study was preregistered at https://osf.io/73xty .

背景:对学术诚信的研究表明,学生的抄袭和作弊率很高。我们开发了一个在线教学工具,诚信游戏(https://integgame.eu/),它使用严肃的游戏来教授学术诚信。在本文中,我们测试了软干预的影响-一个简短的测验-被添加到诚信游戏网站,以提高用户学习诚信的兴趣。基于行为科学的一般原则,我们的测验突出了诚信问题的复杂性,产生了社会比较,并提供了个性化的建议。我们期望这些干预措施会产生对知识的需求,并鼓励参与者花更多的时间在网站上。方法:在瑞士和法国的N = 405名学生的随机对照试验中,一半的用户在玩严肃游戏之前需要做一个简短的测试,而另一半则可以直接玩游戏。我们测量了他们玩游戏的时间,在实验后的调查中,我们测量了他们学习诚信问题的愿望以及他们对诚信问题的理解。结果:与我们的预期相反,测验对玩严肃游戏的时间有负面影响。此外,测验并没有增加参与者学习诚信问题的愿望,也没有增加他们对这个话题的整体理解。结论:我们的测试并没有对好奇心或对诚信问题的理解产生任何可衡量的影响,而且可能会对人们在诚信游戏网站上花费的时间产生负面影响。我们的研究结果强调了在现实世界中实施行为洞察的困难。试验注册:该研究在https://osf.io/73xty上进行了预注册。
{"title":"The quizzical failure of a nudge on academic integrity education: a randomized controlled trial.","authors":"Aurélien Allard, Anna Catharina Vieira Armond, Mads Paludan Goddiksen, Mikkel Willum Johansen, Hillar Loor, Céline Schöpfer, Orsolya Varga, Christine Clavien","doi":"10.1186/s41073-023-00139-z","DOIUrl":"https://doi.org/10.1186/s41073-023-00139-z","url":null,"abstract":"<p><strong>Background: </strong>Studies on academic integrity reveal high rates of plagiarism and cheating among students. We have developed an online teaching tool, Integrity Games ( https://integgame.eu/ ), that uses serious games to teach academic integrity. In this paper, we test the impact of a soft intervention - a short quiz - that was added to the Integrity Games website to increase users' interest in learning about integrity. Based on general principles of behavioral science, our quiz highlighted the intricacy of integrity issues, generated social comparisons, and produced personalized advice. We expected that these interventions would create a need for knowledge and encourage participants to spend more time on the website.</p><p><strong>Methods: </strong>In a randomized controlled trial involving N = 405 students from Switzerland and France, half of the users had to take a short quiz before playing the serious games, while the other half could directly play the games. We measured how much time they spent playing the games, and, in a post-experimental survey, we measured their desire to learn about integrity issues and their understanding of integrity issues.</p><p><strong>Results: </strong>Contrary to our expectations, the quiz had a negative impact on time spent playing the serious games. Moreover, the quiz did not increase participants' desire to learn about integrity issues or their overall understanding of the topic.</p><p><strong>Conclusions: </strong>Our quiz did not have any measurable impact on curiosity or understanding of integrity issues, and may have had a negative impact on time spent on the Integrity games website. Our results highlight the difficulty of implementing behavioral insights in a real-world setting.</p><p><strong>Trial registration: </strong>The study was preregistered at https://osf.io/73xty .</p>","PeriodicalId":74682,"journal":{"name":"Research integrity and peer review","volume":"8 1","pages":"15"},"PeriodicalIF":0.0,"publicationDate":"2023-11-30","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10688455/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"138464957","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 0
Peer reviewers' willingness to review, their recommendations and quality of reviews after the Finnish Medical Journal switched from single-blind to double-blind peer review. 《芬兰医学杂志》从单盲评审改为双盲评审后,同行评审人的评审意愿、他们的建议和评审质量。
Q1 ETHICS Pub Date : 2023-10-24 DOI: 10.1186/s41073-023-00140-6
Piitu Parmanne, Joonas Laajava, Noora Järvinen, Terttu Harju, Mauri Marttunen, Pertti Saloheimo

Background: There is a power imbalance between authors and reviewers in single-blind peer review. We explored how switching from single-blind to double-blind peer review affected 1) the willingness of experts to review, 2) their publication recommendations, and 3) the quality of review reports.

Methods: The Finnish Medical Journal switched from single-blind to double-blind peer review in September 2017. The proportion of review invitations that resulted in a received review report was counted. The reviewers' recommendations of "accept as is", "minor revision", "major revision" or "reject" were explored. The content of the reviews was assessed by two experienced reviewers using the Review Quality Instrument modified to apply to both original research and review manuscripts. The study material comprised reviews submitted from September 2017 to February 2018. The controls were the reviews submitted between September 2015 and February 2016 and between September 2016 and February 2017. The reviewers' recommendations and the scorings of quality assessments were tested with the Chi square test, and the means of quality assessments with the independent-samples t test.

Results: A total of 118 double-blind first-round reviews of 59 manuscripts were compared with 232 single-blind first-round reviews of 116 manuscripts. The proportion of successful review invitations when reviewing single-blinded was 67%, and when reviewing double-blinded, 66%. When reviewing double-blinded, the reviewers recommended accept as is or minor revision less often than during the control period (59% vs. 73%), and major revision or rejection more often (41% vs 27%, P = 0.010). For the quality assessment, 116 reviews from the double-blind period were compared with 104 reviews conducted between September 2016 and February 2017. On a 1-5 scale (1 poor, 5 excellent), double-blind reviews received higher overall proportion of ratings of 4 and 5 than single-blind reviews (56% vs. 49%, P < 0.001). Means for the overall quality of double-blind reviews were 3.38 (IQR, 3.33-3.44) vs. 3.22 (3.17-3.28; P < 0.001) for single-blind reviews.

Conclusions: The quality of the reviews conducted double-blind was better than of those conducted single-blind. Switching to double-blind review did not alter the reviewers' willingness to review. The reviewers became slightly more critical.

背景:在单盲同行评审中,作者和评审者之间存在权力失衡。我们探讨了从单盲到双盲同行评审的转变如何影响1)专家的评审意愿,2)他们的出版建议,以及3)评审报告的质量。方法:2017年9月,《芬兰医学杂志》从单盲同行评审改为双盲同行评审。统计了收到审查报告的审查邀请的比例。对评审员提出的“照原样接受”、“小修改”、“大修改”或“拒绝”的建议进行了探讨。评审内容由两名经验丰富的评审员使用经修改的评审质量工具进行评估,该工具适用于原始研究和评审手稿。研究材料包括2017年9月至2018年2月提交的综述。对照是2015年9月至2016年2月以及2016年9月和2017年2月之间提交的审查。评审员的建议和质量评估的评分采用卡方检验,质量评估的平均值采用独立样本t检验。结果:对59篇稿件的118篇双盲首轮综述与116篇稿件的232篇单盲首轮综述进行了比较。单盲评审时成功评审邀请的比例为67%,双盲评审时为66%。在双盲评审时,评审员建议接受原样或次要修订的频率低于对照期(59%对73%),接受主要修订或拒绝的频率更高(41%对27%,P = 0.010)。对于质量评估,将双盲期的116篇综述与2016年9月至2017年2月期间进行的104篇综述进行了比较。在1-5分制(1分差,5分优)中,双盲评审获得的4分和5分的总体评分比例高于单盲评审(56%对49%,P 结论:双盲评审的质量优于单盲评审。转为双盲审查并没有改变审查者的审查意愿。评审员们变得更加挑剔了。
{"title":"Peer reviewers' willingness to review, their recommendations and quality of reviews after the Finnish Medical Journal switched from single-blind to double-blind peer review.","authors":"Piitu Parmanne, Joonas Laajava, Noora Järvinen, Terttu Harju, Mauri Marttunen, Pertti Saloheimo","doi":"10.1186/s41073-023-00140-6","DOIUrl":"10.1186/s41073-023-00140-6","url":null,"abstract":"<p><strong>Background: </strong>There is a power imbalance between authors and reviewers in single-blind peer review. We explored how switching from single-blind to double-blind peer review affected 1) the willingness of experts to review, 2) their publication recommendations, and 3) the quality of review reports.</p><p><strong>Methods: </strong>The Finnish Medical Journal switched from single-blind to double-blind peer review in September 2017. The proportion of review invitations that resulted in a received review report was counted. The reviewers' recommendations of \"accept as is\", \"minor revision\", \"major revision\" or \"reject\" were explored. The content of the reviews was assessed by two experienced reviewers using the Review Quality Instrument modified to apply to both original research and review manuscripts. The study material comprised reviews submitted from September 2017 to February 2018. The controls were the reviews submitted between September 2015 and February 2016 and between September 2016 and February 2017. The reviewers' recommendations and the scorings of quality assessments were tested with the Chi square test, and the means of quality assessments with the independent-samples t test.</p><p><strong>Results: </strong>A total of 118 double-blind first-round reviews of 59 manuscripts were compared with 232 single-blind first-round reviews of 116 manuscripts. The proportion of successful review invitations when reviewing single-blinded was 67%, and when reviewing double-blinded, 66%. When reviewing double-blinded, the reviewers recommended accept as is or minor revision less often than during the control period (59% vs. 73%), and major revision or rejection more often (41% vs 27%, P = 0.010). For the quality assessment, 116 reviews from the double-blind period were compared with 104 reviews conducted between September 2016 and February 2017. On a 1-5 scale (1 poor, 5 excellent), double-blind reviews received higher overall proportion of ratings of 4 and 5 than single-blind reviews (56% vs. 49%, P < 0.001). Means for the overall quality of double-blind reviews were 3.38 (IQR, 3.33-3.44) vs. 3.22 (3.17-3.28; P < 0.001) for single-blind reviews.</p><p><strong>Conclusions: </strong>The quality of the reviews conducted double-blind was better than of those conducted single-blind. Switching to double-blind review did not alter the reviewers' willingness to review. The reviewers became slightly more critical.</p>","PeriodicalId":74682,"journal":{"name":"Research integrity and peer review","volume":"8 1","pages":"14"},"PeriodicalIF":0.0,"publicationDate":"2023-10-24","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10598992/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"50159492","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 0
Bridges of perspectives: representation of people with lived experience of spinal cord injury in editorial boards and peer review. 视角的桥梁:有脊髓损伤生活经历的人在编辑委员会和同行评审中的代表性。
Q1 ETHICS Pub Date : 2023-09-21 DOI: 10.1186/s41073-023-00138-0
Anna Nuechterlein, Tanya Barretto, Alaa Yehia, Judy Illes

Background: Diversity among editorial boards and in the peer review process maximizes the likelihood that the dissemination of reported results is both relevant and respectful to readers and end users. Past studies have examined diversity among editorial board members and reviewers for factors such as gender, geographic location, and race, but limited research has explored the representation of people with disabilities. Here, we sought to understand the landscape of inclusivity of people with lived experience of spinal cord injury specifically in journals publishing papers (2012-2022) on their quality of life.

Methods: An open and closed 12-question adaptive survey was disseminated to 31 journal editors over a one-month period beginning December 2022.

Results: We received 10 fully completed and 5 partially completed survey responses (response rate 48%). Notwithstanding the small sample, over 50% (8/15) of respondents indicated that their journal review practices involve people with lived experience of spinal cord injury, signaling positive even if incomplete inclusivity practices. The most notable reported barriers to achieving this goal related to identifying and recruiting people with lived experience to serve in the review and editorial process.

Conclusions: In this study we found positive but incomplete trends toward inclusivity in journal practices involving people with lived experience of spinal cord injury. We recommend, therefore, that explicit and genuine efforts are directed toward recruitment through community-based channels. To improve representation even further, we suggest that editors and reviewers be offered the opportunity to self-identify as living with a disability without discrimination or bias.

背景:编辑委员会和同行评审过程中的多样性最大限度地提高了所报告结果的传播对读者和最终用户既相关又尊重的可能性。过去的研究考察了编委会成员和评审员的性别、地理位置和种族等因素的多样性,但对残疾人代表性的研究有限。在这里,我们试图了解有脊髓损伤生活经历的人的包容性,特别是在发表关于他们生活质量的论文(2012-2022)的期刊上。方法:从2022年12月开始,在一个月的时间里,向31名期刊编辑分发了一项开放和封闭的12个问题的适应性调查。结果:我们收到了10份完全完成和5份部分完成的调查回复(回复率48%)。尽管样本很小,但超过50%(8/15)的受访者表示,他们的期刊综述实践涉及有脊髓损伤经历的人,即使包容性实践不完整,这也是积极的。据报道,实现这一目标的最显著障碍是确定和招募有生活经验的人参与审查和编辑过程。结论:在这项研究中,我们发现,在涉及有脊髓损伤经历的人的期刊实践中,有积极但不完全的包容性趋势。因此,我们建议通过社区渠道进行明确和真诚的招聘。为了进一步提高代表性,我们建议编辑和审稿人有机会在没有歧视或偏见的情况下自我认定为残疾人。
{"title":"Bridges of perspectives: representation of people with lived experience of spinal cord injury in editorial boards and peer review.","authors":"Anna Nuechterlein, Tanya Barretto, Alaa Yehia, Judy Illes","doi":"10.1186/s41073-023-00138-0","DOIUrl":"10.1186/s41073-023-00138-0","url":null,"abstract":"<p><strong>Background: </strong>Diversity among editorial boards and in the peer review process maximizes the likelihood that the dissemination of reported results is both relevant and respectful to readers and end users. Past studies have examined diversity among editorial board members and reviewers for factors such as gender, geographic location, and race, but limited research has explored the representation of people with disabilities. Here, we sought to understand the landscape of inclusivity of people with lived experience of spinal cord injury specifically in journals publishing papers (2012-2022) on their quality of life.</p><p><strong>Methods: </strong>An open and closed 12-question adaptive survey was disseminated to 31 journal editors over a one-month period beginning December 2022.</p><p><strong>Results: </strong>We received 10 fully completed and 5 partially completed survey responses (response rate 48%). Notwithstanding the small sample, over 50% (8/15) of respondents indicated that their journal review practices involve people with lived experience of spinal cord injury, signaling positive even if incomplete inclusivity practices. The most notable reported barriers to achieving this goal related to identifying and recruiting people with lived experience to serve in the review and editorial process.</p><p><strong>Conclusions: </strong>In this study we found positive but incomplete trends toward inclusivity in journal practices involving people with lived experience of spinal cord injury. We recommend, therefore, that explicit and genuine efforts are directed toward recruitment through community-based channels. To improve representation even further, we suggest that editors and reviewers be offered the opportunity to self-identify as living with a disability without discrimination or bias.</p>","PeriodicalId":74682,"journal":{"name":"Research integrity and peer review","volume":"8 1","pages":"12"},"PeriodicalIF":0.0,"publicationDate":"2023-09-21","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10512589/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"41159668","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 0
Authorship and citation patterns of highly cited biomedical researchers: a cross-sectional study. 高被引生物医学研究者的作者身份和被引模式:一项横断面研究。
Q1 ETHICS Pub Date : 2023-09-05 DOI: 10.1186/s41073-023-00137-1
Thomas Perneger

Background: Scientific productivity is often evaluated by means of cumulative citation metrics. Different metrics produce different incentives. The H-index assigns full credit from a citation to each coauthor, and thus may encourage multiple collaborations in mid-list author roles. In contrast, the Hm-index assigns only a fraction 1/k of citation credit to each of k coauthors of an article, and thus may encourage research done by smaller teams, and in first or last author roles. Whether H and Hm indices are influenced by different authorship patterns has not been examined.

Methods: Using a publicly available Scopus database, I examined associations between the numbers of research articles published as single, first, mid-list, or last author between 1990 and 2019, and the H-index and the Hm-index, among 18,231 leading researchers in the health sciences.

Results: Adjusting for career duration and other article types, the H-index was negatively associated with the number of single author articles (partial Pearson r -0.06) and first author articles (-0.08), but positively associated with the number of mid-list (0.64) and last author articles (0.21). In contrast, all associations were positive for the Hm-index (0.04 for single author articles, 0.18 for first author articles, 0.24 for mid-list articles, and 0.46 for last author articles).

Conclusion: The H-index and the Hm-index do not reflect the same authorship patterns: the full-credit H-index is predominantly associated with mid-list authorship, whereas the partial-credit Hm-index is driven by more balanced publication patterns, and is most strongly associated with last-author articles. Since performance metrics may act as incentives, the selection of a citation metric should receive careful consideration.

背景:科学生产力通常通过累积引用指标来评估。不同的指标产生不同的激励。h指数将引文的全部荣誉分配给每个共同作者,因此可能会鼓励以中等作者角色进行多次合作。相比之下,hm指数只给一篇文章的k个共同作者分配了1/k的引用信用,因此可能会鼓励较小的团队进行研究,并以第一或最后作者的身份进行研究。H和Hm指数是否受到不同作者模式的影响尚未得到检验。方法:使用公开可用的Scopus数据库,我检查了1990年至2019年期间以单一作者、第一作者、中作者或最后作者发表的研究文章数量与h指数和hm指数之间的关系,研究对象是18231名健康科学领域的主要研究人员。结果:调整职业时间和其他文章类型后,h指数与单作者文章数(偏Pearson r -0.06)和第一作者文章数(-0.08)呈负相关,与中位作者文章数(0.64)和末位作者文章数(0.21)呈正相关。相比之下,所有相关的hm指数都是正的(单作者文章为0.04,第一作者文章为0.18,中间列表文章为0.24,最后作者文章为0.46)。结论:H-index和Hm-index并不反映相同的作者模式:完全署名的H-index主要与中排作者相关,而部分署名的H-index受更平衡的发表模式驱动,与最后作者的文章关系最密切。由于绩效指标可能起到激励作用,因此应该仔细考虑引用指标的选择。
{"title":"Authorship and citation patterns of highly cited biomedical researchers: a cross-sectional study.","authors":"Thomas Perneger","doi":"10.1186/s41073-023-00137-1","DOIUrl":"10.1186/s41073-023-00137-1","url":null,"abstract":"<p><strong>Background: </strong>Scientific productivity is often evaluated by means of cumulative citation metrics. Different metrics produce different incentives. The H-index assigns full credit from a citation to each coauthor, and thus may encourage multiple collaborations in mid-list author roles. In contrast, the Hm-index assigns only a fraction 1/k of citation credit to each of k coauthors of an article, and thus may encourage research done by smaller teams, and in first or last author roles. Whether H and Hm indices are influenced by different authorship patterns has not been examined.</p><p><strong>Methods: </strong>Using a publicly available Scopus database, I examined associations between the numbers of research articles published as single, first, mid-list, or last author between 1990 and 2019, and the H-index and the Hm-index, among 18,231 leading researchers in the health sciences.</p><p><strong>Results: </strong>Adjusting for career duration and other article types, the H-index was negatively associated with the number of single author articles (partial Pearson r -0.06) and first author articles (-0.08), but positively associated with the number of mid-list (0.64) and last author articles (0.21). In contrast, all associations were positive for the Hm-index (0.04 for single author articles, 0.18 for first author articles, 0.24 for mid-list articles, and 0.46 for last author articles).</p><p><strong>Conclusion: </strong>The H-index and the Hm-index do not reflect the same authorship patterns: the full-credit H-index is predominantly associated with mid-list authorship, whereas the partial-credit Hm-index is driven by more balanced publication patterns, and is most strongly associated with last-author articles. Since performance metrics may act as incentives, the selection of a citation metric should receive careful consideration.</p>","PeriodicalId":74682,"journal":{"name":"Research integrity and peer review","volume":"8 1","pages":"13"},"PeriodicalIF":0.0,"publicationDate":"2023-09-05","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10478343/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"10159698","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 0
Reporting quality of abstracts and inconsistencies with full text articles in pediatric orthopedic publications. 小儿骨科出版物中摘要的报告质量和与全文文章的不一致。
Q1 ETHICS Pub Date : 2023-08-23 DOI: 10.1186/s41073-023-00135-3
Sherif Ahmed Kamel, Tamer A El-Sobky

Background: Abstracts should provide a brief yet comprehensive reporting of all components of a manuscript. Inaccurate reporting may mislead readers and impact citation practices. It was our goal to investigate the reporting quality of abstracts of interventional observational studies in three major pediatric orthopedic journals and to analyze any reporting inconsistencies between those abstracts and their corresponding full-text articles.

Methods: We selected a sample of 55 abstracts and their full-text articles published between 2018 and 2022. Included articles were primary therapeutic research investigating the results of treatments or interventions. Abstracts were scrutinized for reporting quality and inconsistencies with their full-text versions with a 22-itemized checklist. The reporting quality of titles was assessed by a 3-items categorical scale.

Results: In 48 (87%) of articles there were abstract reporting inaccuracies related to patient demographics. The study's follow-up and complications were not reported in 21 (38%) of abstracts each. Most common inconsistencies between the abstracts and full-text articles were related to reporting of inclusion or exclusion criteria in 39 (71%) and study correlations in 27 (49%) of articles. Reporting quality of the titles was insufficient in 33 (60%) of articles.

Conclusions: In our study we found low reporting quality of abstracts and noticeable inconsistencies with full-text articles, especially regarding inclusion or exclusion criteria and study correlations. While the current sample is likely not representative of overall pediatric orthopedic literature, we recommend that authors, reviewers, and editors ensure abstracts are reported accurately, ideally following the appropriate reporting guidelines, and that they double check that there are no inconsistencies between abstracts and full text articles. To capture essential study information, journals should also consider increasing abstract word limits.

背景:摘要应该提供一个简短而全面的报告的所有组成部分的手稿。不准确的报道可能会误导读者并影响引用实践。我们的目的是调查三个主要儿科骨科期刊的介入观察性研究摘要的报道质量,并分析这些摘要与其相应的全文文章之间的任何报道不一致之处。方法:选取2018 - 2022年间发表的55篇摘要及其全文。纳入的文章是调查治疗或干预结果的初步治疗研究。摘要通过22项清单审查报告质量和与全文版本的不一致之处。题目的报告质量采用3项分类量表进行评估。结果:48篇(87%)文章的摘要报告与患者人口统计学相关的不准确。21篇(38%)摘要未报道该研究的随访和并发症。摘要和全文文章之间最常见的不一致与39篇(71%)文章的纳入或排除标准报告和27篇(49%)文章的研究相关性报告有关。33篇(60%)文章标题报告质量不足。结论:在我们的研究中,我们发现摘要的报告质量较低,并且与全文文章存在明显的不一致,特别是在纳入或排除标准和研究相关性方面。虽然目前的样本可能不能代表整个儿科骨科文献,但我们建议作者、审稿人和编辑确保摘要报告准确,理想情况下遵循适当的报告指南,并仔细检查摘要与全文文章之间没有不一致之处。为了获取重要的研究信息,期刊还应考虑增加抽象字数限制。
{"title":"Reporting quality of abstracts and inconsistencies with full text articles in pediatric orthopedic publications.","authors":"Sherif Ahmed Kamel, Tamer A El-Sobky","doi":"10.1186/s41073-023-00135-3","DOIUrl":"10.1186/s41073-023-00135-3","url":null,"abstract":"<p><strong>Background: </strong>Abstracts should provide a brief yet comprehensive reporting of all components of a manuscript. Inaccurate reporting may mislead readers and impact citation practices. It was our goal to investigate the reporting quality of abstracts of interventional observational studies in three major pediatric orthopedic journals and to analyze any reporting inconsistencies between those abstracts and their corresponding full-text articles.</p><p><strong>Methods: </strong>We selected a sample of 55 abstracts and their full-text articles published between 2018 and 2022. Included articles were primary therapeutic research investigating the results of treatments or interventions. Abstracts were scrutinized for reporting quality and inconsistencies with their full-text versions with a 22-itemized checklist. The reporting quality of titles was assessed by a 3-items categorical scale.</p><p><strong>Results: </strong>In 48 (87%) of articles there were abstract reporting inaccuracies related to patient demographics. The study's follow-up and complications were not reported in 21 (38%) of abstracts each. Most common inconsistencies between the abstracts and full-text articles were related to reporting of inclusion or exclusion criteria in 39 (71%) and study correlations in 27 (49%) of articles. Reporting quality of the titles was insufficient in 33 (60%) of articles.</p><p><strong>Conclusions: </strong>In our study we found low reporting quality of abstracts and noticeable inconsistencies with full-text articles, especially regarding inclusion or exclusion criteria and study correlations. While the current sample is likely not representative of overall pediatric orthopedic literature, we recommend that authors, reviewers, and editors ensure abstracts are reported accurately, ideally following the appropriate reporting guidelines, and that they double check that there are no inconsistencies between abstracts and full text articles. To capture essential study information, journals should also consider increasing abstract word limits.</p>","PeriodicalId":74682,"journal":{"name":"Research integrity and peer review","volume":"8 1","pages":"11"},"PeriodicalIF":0.0,"publicationDate":"2023-08-23","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10463470/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"10121003","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 0
Raising concerns on questionable ethics approvals - a case study of 456 trials from the Institut Hospitalo-Universitaire Méditerranée Infection. 对有问题的伦理批准提出关注——一项对来自法国医院-大学(Institut hospital - universitaire)的456项试验的案例研究。
Q1 ETHICS Pub Date : 2023-08-03 DOI: 10.1186/s41073-023-00134-4
Fabrice Frank, Nans Florens, Gideon Meyerowitz-Katz, Jérôme Barriere, Éric Billy, Véronique Saada, Alexander Samuel, Jacques Robert, Lonni Besançon

Background: The practice of clinical research is strictly regulated by law. During submission and review processes, compliance of such research with the laws enforced in the country where it was conducted is not always correctly filled in by the authors or verified by the editors. Here, we report a case of a single institution for which one may find hundreds of publications with seemingly relevant ethical concerns, along with 10 months of follow-up through contacts with the editors of these articles. We thus argue for a stricter control of ethical authorization by scientific editors and we call on publishers to cooperate to this end.

Methods: We present an investigation of the ethics and legal aspects of 456 studies published by the IHU-MI (Institut Hospitalo-Universitaire Méditerranée Infection) in Marseille, France.

Results: We identified a wide range of issues with the stated research authorization and ethics of the published studies with respect to the Institutional Review Board and the approval presented. Among the studies investigated, 248 were conducted with the same ethics approval number, even though the subjects, samples, and countries of investigation were different. Thirty-nine (39) did not even contain a reference to the ethics approval number while they present research on human beings. We thus contacted the journals that published these articles and provide their responses to our concerns. It should be noted that, since our investigation and reporting to journals, PLOS has issued expressions of concerns for several publications we analyze here.

Conclusion: This case presents an investigation of the veracity of ethical approval, and more than 10 months of follow-up by independent researchers. We call for stricter control and cooperation in handling of these cases, including editorial requirement to upload ethical approval documents, guidelines from COPE to address such ethical concerns, and transparent editorial policies and timelines to answer such concerns. All supplementary materials are available.

背景:临床研究实践受到法律的严格规范。在提交和审查过程中,作者并不总是正确填写或编辑核实这些研究是否符合进行研究的国家所执行的法律。在这里,我们报告了一个单一机构的案例,人们可能会发现数百篇看似相关的伦理问题的出版物,以及通过与这些文章的编辑联系的10个月的随访。因此,我们主张对科学编辑的伦理授权进行更严格的控制,并呼吁出版商为此进行合作。方法:我们对法国马赛IHU-MI(医院-大学研究所)发表的456项研究的伦理和法律方面进行了调查。结果:我们发现了与机构审查委员会和提交的批准有关的已发表研究的研究授权和伦理方面的广泛问题。在被调查的研究中,尽管研究对象、样本和调查国家不同,但有248项研究使用了相同的伦理批准号。39项研究在展示人体研究时甚至没有提及伦理批准号。因此,我们联系了发表这些文章的期刊,并提供了他们对我们关注的问题的回应。应该指出的是,自从我们的调查和向期刊报告以来,PLOS已经发布了我们在这里分析的几种出版物的担忧表达。结论:本病例对伦理批准的真实性进行了调查,并由独立研究人员进行了10个多月的随访。我们呼吁在处理这些案件时进行更严格的控制和合作,包括编辑要求上传伦理批准文件,COPE的指导方针来解决这些伦理问题,以及透明的编辑政策和时间表来回答这些问题。所有补充材料都准备好了。
{"title":"Raising concerns on questionable ethics approvals - a case study of 456 trials from the Institut Hospitalo-Universitaire Méditerranée Infection.","authors":"Fabrice Frank,&nbsp;Nans Florens,&nbsp;Gideon Meyerowitz-Katz,&nbsp;Jérôme Barriere,&nbsp;Éric Billy,&nbsp;Véronique Saada,&nbsp;Alexander Samuel,&nbsp;Jacques Robert,&nbsp;Lonni Besançon","doi":"10.1186/s41073-023-00134-4","DOIUrl":"https://doi.org/10.1186/s41073-023-00134-4","url":null,"abstract":"<p><strong>Background: </strong>The practice of clinical research is strictly regulated by law. During submission and review processes, compliance of such research with the laws enforced in the country where it was conducted is not always correctly filled in by the authors or verified by the editors. Here, we report a case of a single institution for which one may find hundreds of publications with seemingly relevant ethical concerns, along with 10 months of follow-up through contacts with the editors of these articles. We thus argue for a stricter control of ethical authorization by scientific editors and we call on publishers to cooperate to this end.</p><p><strong>Methods: </strong>We present an investigation of the ethics and legal aspects of 456 studies published by the IHU-MI (Institut Hospitalo-Universitaire Méditerranée Infection) in Marseille, France.</p><p><strong>Results: </strong>We identified a wide range of issues with the stated research authorization and ethics of the published studies with respect to the Institutional Review Board and the approval presented. Among the studies investigated, 248 were conducted with the same ethics approval number, even though the subjects, samples, and countries of investigation were different. Thirty-nine (39) did not even contain a reference to the ethics approval number while they present research on human beings. We thus contacted the journals that published these articles and provide their responses to our concerns. It should be noted that, since our investigation and reporting to journals, PLOS has issued expressions of concerns for several publications we analyze here.</p><p><strong>Conclusion: </strong>This case presents an investigation of the veracity of ethical approval, and more than 10 months of follow-up by independent researchers. We call for stricter control and cooperation in handling of these cases, including editorial requirement to upload ethical approval documents, guidelines from COPE to address such ethical concerns, and transparent editorial policies and timelines to answer such concerns. All supplementary materials are available.</p>","PeriodicalId":74682,"journal":{"name":"Research integrity and peer review","volume":"8 1","pages":"9"},"PeriodicalIF":0.0,"publicationDate":"2023-08-03","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10398994/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"9938883","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 1
A new approach to grant review assessments: score, then rank. 一种授予评审评估的新方法:评分,然后排名。
Q1 ETHICS Pub Date : 2023-07-24 DOI: 10.1186/s41073-023-00131-7
Stephen A Gallo, Michael Pearce, Carole J Lee, Elena A Erosheva

Background: In many grant review settings, proposals are selected for funding on the basis of summary statistics of review ratings. Challenges of this approach (including the presence of ties and unclear ordering of funding preference for proposals) could be mitigated if rankings such as top-k preferences or paired comparisons, which are local evaluations that enforce ordering across proposals, were also collected and incorporated in the analysis of review ratings. However, analyzing ratings and rankings simultaneously has not been done until recently. This paper describes a practical method for integrating rankings and scores and demonstrates its usefulness for making funding decisions in real-world applications.

Methods: We first present the application of our existing joint model for rankings and ratings, the Mallows-Binomial, in obtaining an integrated score for each proposal and generating the induced preference ordering. We then apply this methodology to several theoretical "toy" examples of rating and ranking data, designed to demonstrate specific properties of the model. We then describe an innovative protocol for collecting rankings of the top-six proposals as an add-on to the typical peer review scoring procedures and provide a case study using actual peer review data to exemplify the output and how the model can appropriately resolve judges' evaluations.

Results: For the theoretical examples, we show how the model can provide a preference order to equally rated proposals by incorporating rankings, to proposals using ratings and only partial rankings (and how they differ from a ratings-only approach) and to proposals where judges provide internally inconsistent ratings/rankings and outlier scoring. Finally, we discuss how, using real world panel data, this method can provide information about funding priority with a level of accuracy in a well-suited format for research funding decisions.

Conclusions: A methodology is provided to collect and employ both rating and ranking data in peer review assessments of proposal submission quality, highlighting several advantages over methods relying on ratings alone. This method leverages information to most accurately distill reviewer opinion into a useful output to make an informed funding decision and is general enough to be applied to settings such as in the NIH panel review process.

背景:在许多拨款审查设置中,提案是根据审查评分的汇总统计来选择资助的。如果还收集诸如top-k偏好或配对比较之类的排名,并将其纳入审查评级的分析中,则可以减轻这种方法的挑战(包括存在联系和提案资金偏好的不明确排序)。配对比较是在提案之间强制排序的本地评估。但是,直到最近才同时分析收视率和排名。本文描述了一种整合排名和分数的实用方法,并展示了它在实际应用中做出资助决策的有用性。方法:我们首先介绍了我们现有的排名和评级联合模型,Mallows-Binomial,在获得每个提案的综合得分和生成诱导偏好排序中的应用。然后,我们将这种方法应用于几个评级和排名数据的理论“玩具”示例,旨在展示模型的特定属性。然后,我们描述了一个收集前六名提案排名的创新协议,作为典型同行评议评分程序的附加程序,并提供了一个使用实际同行评议数据的案例研究,以举例说明输出以及该模型如何适当地解决评委的评估。结果:对于理论示例,我们展示了该模型如何通过结合排名来为同等评级的提案提供偏好顺序,如何为使用评级和仅部分排名的提案提供偏好顺序(以及它们与仅评级方法的区别),以及如何为评委提供内部不一致的评级/排名和异常值评分的提案提供偏好顺序。最后,我们讨论了如何使用真实世界的面板数据,这种方法能够以一种非常适合研究资助决策的格式,以一定程度的准确性提供有关资助优先级的信息。结论:提供了一种方法来收集和使用评级和排名数据在提案提交质量的同行评议评估中,突出了仅依赖评级方法的几个优势。这种方法利用信息,最准确地将审稿人的意见提炼成有用的输出,以做出明智的资助决定,并且足够普遍,可以应用于NIH小组审查过程等设置。
{"title":"A new approach to grant review assessments: score, then rank.","authors":"Stephen A Gallo,&nbsp;Michael Pearce,&nbsp;Carole J Lee,&nbsp;Elena A Erosheva","doi":"10.1186/s41073-023-00131-7","DOIUrl":"https://doi.org/10.1186/s41073-023-00131-7","url":null,"abstract":"<p><strong>Background: </strong>In many grant review settings, proposals are selected for funding on the basis of summary statistics of review ratings. Challenges of this approach (including the presence of ties and unclear ordering of funding preference for proposals) could be mitigated if rankings such as top-k preferences or paired comparisons, which are local evaluations that enforce ordering across proposals, were also collected and incorporated in the analysis of review ratings. However, analyzing ratings and rankings simultaneously has not been done until recently. This paper describes a practical method for integrating rankings and scores and demonstrates its usefulness for making funding decisions in real-world applications.</p><p><strong>Methods: </strong>We first present the application of our existing joint model for rankings and ratings, the Mallows-Binomial, in obtaining an integrated score for each proposal and generating the induced preference ordering. We then apply this methodology to several theoretical \"toy\" examples of rating and ranking data, designed to demonstrate specific properties of the model. We then describe an innovative protocol for collecting rankings of the top-six proposals as an add-on to the typical peer review scoring procedures and provide a case study using actual peer review data to exemplify the output and how the model can appropriately resolve judges' evaluations.</p><p><strong>Results: </strong>For the theoretical examples, we show how the model can provide a preference order to equally rated proposals by incorporating rankings, to proposals using ratings and only partial rankings (and how they differ from a ratings-only approach) and to proposals where judges provide internally inconsistent ratings/rankings and outlier scoring. Finally, we discuss how, using real world panel data, this method can provide information about funding priority with a level of accuracy in a well-suited format for research funding decisions.</p><p><strong>Conclusions: </strong>A methodology is provided to collect and employ both rating and ranking data in peer review assessments of proposal submission quality, highlighting several advantages over methods relying on ratings alone. This method leverages information to most accurately distill reviewer opinion into a useful output to make an informed funding decision and is general enough to be applied to settings such as in the NIH panel review process.</p>","PeriodicalId":74682,"journal":{"name":"Research integrity and peer review","volume":"8 1","pages":"10"},"PeriodicalIF":0.0,"publicationDate":"2023-07-24","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10367367/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"9865500","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 0
Institutional capacity to prevent and manage research misconduct: perspectives from Kenyan research regulators. 预防和管理科研不端行为的机构能力:来自肯尼亚科研监管机构的观点。
Q1 ETHICS Pub Date : 2023-07-12 DOI: 10.1186/s41073-023-00132-6
Edwin Were, Jepchirchir Kiplagat, Eunice Kaguiri, Rose Ayikukwei, Violet Naanyu

Background: Research misconduct i.e. fabrication, falsification, and plagiarism is associated with individual, institutional, national, and global factors. Researchers' perceptions of weak or non-existent institutional guidelines on the prevention and management of research misconduct can encourage these practices. Few countries in Africa have clear guidance on research misconduct. In Kenya, the capacity to prevent or manage research misconduct in academic and research institutions has not been documented. The objective of this study was to explore the perceptions of Kenyan research regulators on the occurrence of and institutional capacity to prevent or manage research misconduct.

Methods: Interviews with open-ended questions were conducted with 27 research regulators (chairs and secretaries of ethics committees, research directors of academic and research institutions, and national regulatory bodies). Among other questions, participants were asked: (1) How common is research misconduct in your view? (2) Does your institution have the capacity to prevent research misconduct? (3) Does your institution have the capacity to manage research misconduct? Their responses were audiotaped, transcribed, and coded using NVivo software. Deductive coding covered predefined themes including perceptions on occurrence, prevention detection, investigation, and management of research misconduct. Results are presented with illustrative quotes.

Results: Respondents perceived research misconduct to be very common among students developing thesis reports. Their responses suggested there was no dedicated capacity to prevent or manage research misconduct at the institutional and national levels. There were no specific national guidelines on research misconduct. At the institutional level, the only capacity/efforts mentioned were directed at reducing, detecting, and managing student plagiarism. There was no direct mention of the capacity to manage fabrication and falsification or misconduct by faculty researchers. We recommend the development of Kenya code of conduct or research integrity guidelines that would cover misconduct.

背景:研究不端行为,即捏造、伪造和抄袭与个人、机构、国家和全球因素有关。研究人员对预防和管理研究不端行为的薄弱或不存在的机构指导方针的认识可能会鼓励这些做法。非洲很少有国家对研究不端行为有明确的指导。在肯尼亚,学术和研究机构预防或管理研究不端行为的能力尚未得到记录。本研究的目的是探讨肯尼亚研究监管机构对研究不端行为的发生和机构预防或管理研究不端行为的能力的看法。方法:对27名研究监管人员(伦理委员会主席和秘书、学术和研究机构的研究主任以及国家监管机构)进行开放式访谈。在其他问题中,参与者被问到:(1)在你看来,科研不端行为有多普遍?(2)贵机构是否有能力防止研究不端行为?(3)贵机构是否有能力管理研究不端行为?他们的回答被录音,转录,并使用NVivo软件编码。演绎编码涵盖了预定义的主题,包括对研究不端行为的发生、预防、检测、调查和管理的看法。结果给出了说明性引用。结果:受访者认为研究不端行为是非常普遍的学生发展论文报告。他们的答复表明,在机构和国家层面没有专门的能力来预防或管理研究不端行为。没有针对研究不端行为的具体国家指导方针。在机构层面,提到的唯一能力/努力是针对减少、发现和管理学生抄袭。没有直接提到管理捏造、伪造或教员研究人员不当行为的能力。我们建议制定肯尼亚行为准则或研究诚信准则,以涵盖不端行为。
{"title":"Institutional capacity to prevent and manage research misconduct: perspectives from Kenyan research regulators.","authors":"Edwin Were,&nbsp;Jepchirchir Kiplagat,&nbsp;Eunice Kaguiri,&nbsp;Rose Ayikukwei,&nbsp;Violet Naanyu","doi":"10.1186/s41073-023-00132-6","DOIUrl":"https://doi.org/10.1186/s41073-023-00132-6","url":null,"abstract":"<p><strong>Background: </strong>Research misconduct i.e. fabrication, falsification, and plagiarism is associated with individual, institutional, national, and global factors. Researchers' perceptions of weak or non-existent institutional guidelines on the prevention and management of research misconduct can encourage these practices. Few countries in Africa have clear guidance on research misconduct. In Kenya, the capacity to prevent or manage research misconduct in academic and research institutions has not been documented. The objective of this study was to explore the perceptions of Kenyan research regulators on the occurrence of and institutional capacity to prevent or manage research misconduct.</p><p><strong>Methods: </strong>Interviews with open-ended questions were conducted with 27 research regulators (chairs and secretaries of ethics committees, research directors of academic and research institutions, and national regulatory bodies). Among other questions, participants were asked: (1) How common is research misconduct in your view? (2) Does your institution have the capacity to prevent research misconduct? (3) Does your institution have the capacity to manage research misconduct? Their responses were audiotaped, transcribed, and coded using NVivo software. Deductive coding covered predefined themes including perceptions on occurrence, prevention detection, investigation, and management of research misconduct. Results are presented with illustrative quotes.</p><p><strong>Results: </strong>Respondents perceived research misconduct to be very common among students developing thesis reports. Their responses suggested there was no dedicated capacity to prevent or manage research misconduct at the institutional and national levels. There were no specific national guidelines on research misconduct. At the institutional level, the only capacity/efforts mentioned were directed at reducing, detecting, and managing student plagiarism. There was no direct mention of the capacity to manage fabrication and falsification or misconduct by faculty researchers. We recommend the development of Kenya code of conduct or research integrity guidelines that would cover misconduct.</p>","PeriodicalId":74682,"journal":{"name":"Research integrity and peer review","volume":"8 1","pages":"8"},"PeriodicalIF":0.0,"publicationDate":"2023-07-12","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10337100/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"10190722","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 0
期刊
Research integrity and peer review
全部 Acc. Chem. Res. ACS Applied Bio Materials ACS Appl. Electron. Mater. ACS Appl. Energy Mater. ACS Appl. Mater. Interfaces ACS Appl. Nano Mater. ACS Appl. Polym. Mater. ACS BIOMATER-SCI ENG ACS Catal. ACS Cent. Sci. ACS Chem. Biol. ACS Chemical Health & Safety ACS Chem. Neurosci. ACS Comb. Sci. ACS Earth Space Chem. ACS Energy Lett. ACS Infect. Dis. ACS Macro Lett. ACS Mater. Lett. ACS Med. Chem. Lett. ACS Nano ACS Omega ACS Photonics ACS Sens. ACS Sustainable Chem. Eng. ACS Synth. Biol. Anal. Chem. BIOCHEMISTRY-US Bioconjugate Chem. BIOMACROMOLECULES Chem. Res. Toxicol. Chem. Rev. Chem. Mater. CRYST GROWTH DES ENERG FUEL Environ. Sci. Technol. Environ. Sci. Technol. Lett. Eur. J. Inorg. Chem. IND ENG CHEM RES Inorg. Chem. J. Agric. Food. Chem. J. Chem. Eng. Data J. Chem. Educ. J. Chem. Inf. Model. J. Chem. Theory Comput. J. Med. Chem. J. Nat. Prod. J PROTEOME RES J. Am. Chem. Soc. LANGMUIR MACROMOLECULES Mol. Pharmaceutics Nano Lett. Org. Lett. ORG PROCESS RES DEV ORGANOMETALLICS J. Org. Chem. J. Phys. Chem. J. Phys. Chem. A J. Phys. Chem. B J. Phys. Chem. C J. Phys. Chem. Lett. Analyst Anal. Methods Biomater. Sci. Catal. Sci. Technol. Chem. Commun. Chem. Soc. Rev. CHEM EDUC RES PRACT CRYSTENGCOMM Dalton Trans. Energy Environ. Sci. ENVIRON SCI-NANO ENVIRON SCI-PROC IMP ENVIRON SCI-WAT RES Faraday Discuss. Food Funct. Green Chem. Inorg. Chem. Front. Integr. Biol. J. Anal. At. Spectrom. J. Mater. Chem. A J. Mater. Chem. B J. Mater. Chem. C Lab Chip Mater. Chem. Front. Mater. Horiz. MEDCHEMCOMM Metallomics Mol. Biosyst. Mol. Syst. Des. Eng. Nanoscale Nanoscale Horiz. Nat. Prod. Rep. New J. Chem. Org. Biomol. Chem. Org. Chem. Front. PHOTOCH PHOTOBIO SCI PCCP Polym. Chem.
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
0
微信
客服QQ
Book学术公众号 扫码关注我们
反馈
×
意见反馈
请填写您的意见或建议
请填写您的手机或邮箱
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
现在去查看 取消
×
提示
确定
Book学术官方微信
Book学术文献互助
Book学术文献互助群
群 号:481959085
Book学术
文献互助 智能选刊 最新文献 互助须知 联系我们:info@booksci.cn
Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。
Copyright © 2023 Book学术 All rights reserved.
ghs 京公网安备 11010802042870号 京ICP备2023020795号-1