Research integrity and peer review最新文献

Background: Vast sums are distributed based on grant peer review, but studies show that interrater reliability is often low. In this study, we tested the effect of receiving two short individual feedback reports compared to one short general feedback report on the agreement between reviewers.

Methods: A total of 42 reviewers at the Norwegian Foundation Dam were randomly assigned to receive either a general feedback report or an individual feedback report. The general feedback group received one report before the start of the reviews that contained general information about the previous call in which the reviewers participated. In the individual feedback group, the reviewers received two reports, one before the review period (based on the previous call) and one during the period (based on the current call). In the individual feedback group, the reviewers were presented with detailed information on their scoring compared with the review committee as a whole, both before and during the review period. The main outcomes were the proportion of agreement in the eligibility assessment and the average difference in scores between pairs of reviewers assessing the same proposal. The outcomes were measured in 2017 and after the feedback was provided in 2018.

Results: A total of 2398 paired reviews were included in the analysis. There was a significant difference between the two groups in the proportion of absolute agreement on whether the proposal was eligible for the funding programme, with the general feedback group demonstrating a higher rate of agreement. There was no difference between the two groups in terms of the average score difference. However, the agreement regarding the proposal score remained critically low for both groups.

Conclusions: We did not observe changes in proposal score agreement between 2017 and 2018 in reviewers receiving different feedback. The low levels of agreement remain a major concern in grant peer review, and research to identify contributing factors as well as the development and testing of interventions to increase agreement rates are still needed.

Trial registration: The study was preregistered at OSF.io/n4fq3 .

Background: Australian health and medical research funders support substantial research efforts, and incentives within grant funding schemes influence researcher behaviour. We aimed to determine to what extent Australian health and medical funders incentivise responsible research practices.

Methods: We conducted an audit of instructions from research grant and fellowship schemes. Eight national research grants and fellowships were purposively sampled to select schemes that awarded the largest amount of funds. The funding scheme instructions were assessed against 9 criteria to determine to what extent they incentivised these responsible research and reporting practices: (1) publicly register study protocols before starting data collection, (2) register analysis protocols before starting data analysis, (3) make study data openly available, (4) make analysis code openly available, (5) make research materials openly available, (6) discourage use of publication metrics, (7) conduct quality research (e.g. adhere to reporting guidelines), (8) collaborate with a statistician, and (9) adhere to other responsible research practices. Each criterion was answered using one of the following responses: "Instructed", "Encouraged", or "No mention".

Results: Across the 8 schemes from 5 funders, applicants were instructed or encouraged to address a median of 4 (range 0 to 5) of the 9 criteria. Three criteria received no mention in any scheme (register analysis protocols, make analysis code open, collaborate with a statistician). Importantly, most incentives did not seem strong as applicants were only instructed to register study protocols, discourage use of publication metrics and conduct quality research. Other criteria were encouraged but were not required.

Conclusions: Funders could strengthen the incentives for responsible research practices by requiring grant and fellowship applicants to implement these practices in their proposals. Administering institutions could be required to implement these practices to be eligible for funding. Strongly rewarding researchers for implementing robust research practices could lead to sustained improvements in the quality of health and medical research.

Background: The Transparency and Openness Promotion (TOP) Guidelines describe modular standards that journals can adopt to promote open science. The TOP Factor is a metric to describe the extent to which journals have adopted the TOP Guidelines in their policies. Systematic methods and rating instruments are needed to calculate the TOP Factor. Moreover, implementation of these open science policies depends on journal procedures and practices, for which TOP provides no standards or rating instruments.

Methods: We describe a process for assessing journal policies, procedures, and practices according to the TOP Guidelines. We developed this process as part of the Transparency of Research Underpinning Social Intervention Tiers (TRUST) Initiative to advance open science in the social intervention research ecosystem. We also provide new instruments for rating journal instructions to authors (policies), manuscript submission systems (procedures), and published articles (practices) according to standards in the TOP Guidelines. In addition, we describe how to determine the TOP Factor score for a journal, calculate reliability of journal ratings, and assess coherence among a journal's policies, procedures, and practices. As a demonstration of this process, we describe a protocol for studying approximately 345 influential journals that have published research used to inform evidence-based policy.

Discussion: The TRUST Process includes systematic methods and rating instruments for assessing and facilitating implementation of the TOP Guidelines by journals across disciplines. Our study of journals publishing influential social intervention research will provide a comprehensive account of whether these journals have policies, procedures, and practices that are consistent with standards for open science and thereby facilitate the publication of trustworthy findings to inform evidence-based policy. Through this demonstration, we expect to identify ways to refine the TOP Guidelines and the TOP Factor. Refinements could include: improving templates for adoption in journal instructions to authors, manuscript submission systems, and published articles; revising explanatory guidance intended to enhance the use, understanding, and dissemination of the TOP Guidelines; and clarifying the distinctions among different levels of implementation. Research materials are available on the Open Science Framework: https://osf.io/txyr3/ .

Background: Healthcare professionals are exposed to advertisements for prescription drugs in medical journals. Such advertisements may increase prescriptions of new drugs at the expense of older treatments even when they have no added benefits, are more harmful, and are more expensive. The publication of medical advertisements therefore raises ethical questions related to editorial integrity.

Methods: We conducted a descriptive cross-sectional study of all medical advertisements published in the Journal of the Danish Medical Association in 2015. Drugs advertised 6 times or more were compared with older comparators: (1) comparative evidence of added benefit; (2) Defined Daily Dose cost; (3) regulatory safety announcements; and (4) completed and ongoing post-marketing studies 3 years after advertising.

Results: We found 158 medical advertisements for 35 prescription drugs published in 24 issues during 2015, with a median of 7 advertisements per issue (range 0 to 11). Four drug groups and 5 single drugs were advertised 6 times or more, for a total of 10 indications, and we made 14 comparisons with older treatments. We found: (1) 'no added benefit' in 4 (29%) of 14 comparisons, 'uncertain benefits' in 7 (50%), and 'no evidence' in 3 (21%) comparisons. In no comparison did we find evidence of 'substantial added benefit' for the new drug; (2) advertised drugs were 2 to 196 times (median 6) more expensive per Defined Daily Dose; (3) 11 safety announcements for five advertised drugs were issued compared to one announcement for one comparator drug; (4) 20 post-marketing studies (7 completed, 13 ongoing) were requested for the advertised drugs versus 10 studies (4 completed, 6 ongoing) for the comparator drugs, and 7 studies (2 completed, 5 ongoing) assessed both an advertised and a comparator drug at 3 year follow-up.

Conclusions and relevance: In this cross-sectional study of medical advertisements published in the Journal of the Danish Medical Association during 2015, the most advertised drugs did not have documented substantial added benefits over older treatments, whereas they were substantially more expensive. From January 2021, the Journal of the Danish Medical Association no longer publishes medical advertisements.

Background: Concerns about research misbehavior in academic science have sparked interest in the factors that may explain research misbehavior. Often three clusters of factors are distinguished: individual factors, climate factors and publication factors. Our research question was: to what extent can individual, climate and publication factors explain the variance in frequently perceived research misbehaviors?

Methods: From May 2017 until July 2017, we conducted a survey study among academic researchers in Amsterdam. The survey included three measurement instruments that we previously reported individual results of and here we integrate these findings.

Results: One thousand two hundred ninety-eight researchers completed the survey (response rate: 17%). Results showed that individual, climate and publication factors combined explained 34% of variance in perceived frequency of research misbehavior. Individual factors explained 7%, climate factors explained 22% and publication factors 16%.

Conclusions: Our results suggest that the perceptions of the research climate play a substantial role in explaining variance in research misbehavior. This suggests that efforts to improve departmental norms might have a salutary effect on behavior.

Background: Inaccurate, false or incomplete research publications may mislead readers including researchers and decision-makers. It is therefore important that such problems are identified and rectified promptly. This usually involves collaboration between the research institutions and academic journals involved, but these interactions can be problematic.

Methods: These recommendations were developed following discussions at World Conferences on Research Integrity in 2013 and 2017, and at a specially convened 3-day workshop in 2016 involving participants from 7 countries with expertise in publication ethics and research integrity. The recommendations aim to address issues surrounding cooperation and liaison between institutions (e.g. universities) and journals about possible and actual problems with the integrity of reported research arising before and after publication.

Results: The main recommendations are that research institutions should: 1) develop mechanisms for assessing the integrity of reported research (if concerns are raised) that are distinct from processes to determine whether individual researchers have committed misconduct; 2) release relevant sections of reports of research integrity or misconduct investigations to all journals that have published research that was investigated; 3) take responsibility for research performed under their auspices regardless of whether the researcher still works at that institution or how long ago the work was done; 4) work with funders to ensure essential research data is retained for at least 10 years. Journals should: 1) respond to institutions about research integrity cases in a timely manner; 2) have criteria for determining whether, and what type of, information and evidence relating to the integrity of research reports should be passed on to institutions; 3) pass on research integrity concerns to institutions, regardless of whether they intend to accept the work for publication; 4) retain peer review records for at least 10 years to enable the investigation of peer review manipulation or other inappropriate behaviour by authors or reviewers.

Conclusions: Various difficulties can prevent effective cooperation between academic journals and research institutions about research integrity concerns and hinder the correction of the research record if problems are discovered. While the issues and their solutions may vary across different settings, we encourage research institutions, journals and funders to consider how they might improve future collaboration and cooperation on research integrity cases.

Background: A proposal to encourage the preregistration of research on research integrity was developed and adopted as the Amsterdam Agenda at the 5th World Conference on Research Integrity (Amsterdam, 2017). This paper reports on the degree to which abstracts of the 6th World Conference in Research Integrity (Hong Kong, 2019) reported on preregistered research.

Methods: Conference registration data on participants presenting a paper or a poster at 6th WCRI were made available to the research team. Because the data set was too small for inferential statistics this report is limited to a basic description of results and some recommendations that should be considered when taking further steps to improve preregistration.

Results: 19% of the 308 presenters preregistered their research. Of the 56 usable cases, less than half provided information on the six key elements of the Amsterdam Agenda. Others provided information that invalidated their data, such as an uninformative URL. There was no discernable difference between qualitative and quantitative research.

Conclusions: Some presenters at the WCRI have preregistered their research on research integrity, but further steps are needed to increase frequency and completeness of preregistration. One approach to increase preregistration would be to make it a requirement for research presented at the World Conferences on Research Integrity.

Our recent paper ( https://doi.org/10.1186/s41073-020-00096-x ) reported that 43% of reviewer comment sets (n=1491) shared with authors contained at least one unprofessional comment or an incomplete, inaccurate of unsubstantiated critique (IIUC). Publication of this work sparked an online (i.e., Twitter, Instagram, Facebook, and Reddit) conversation surrounding professionalism in peer review. We collected and analyzed these social media comments as they offered real-time responses to our work and provided insight into the views held by commenters and potential peer-reviewers that would be difficult to quantify using existing empirical tools (96 comments from July 24th to September 3rd, 2020). Overall, 75% of comments were positive, of which 59% were supportive and 16% shared similar personal experiences. However, a subset of negative comments emerged (22% of comments were negative and 6% were an unsubstantiated critique of the methodology), that provided potential insight into the reasons underlying unprofessional comments were made during the peer-review process. These comments were classified into three main themes: (1) forced niceness will adversely impact the peer-review process and allow for publication of poor-quality science (5% of online comments); (2) dismissing comments as not offensive to another person because they were not deemed personally offensive to the reader (6%); and (3) authors brought unprofessional comments upon themselves as they submitted substandard work (5%). Here, we argue against these themes as justifications for directing unprofessional comments towards authors during the peer review process. We argue that it is possible to be both critical and professional, and that no author deserves to be the recipient of demeaning ad hominem attacks regardless of supposed provocation. Suggesting otherwise only serves to propagate a toxic culture within peer review. While we previously postulated that establishing a peer-reviewer code of conduct could help improve the peer-review system, we now posit that priority should be given to repairing the negative cultural zeitgeist that exists in peer-review.

Background: Scientists communicate progress and exchange information via publication and presentation at scientific meetings. We previously showed that text similarity analysis applied to Medline can identify and quantify plagiarism and duplicate publications in peer-reviewed biomedical journals. In the present study, we applied the same analysis to a large sample of conference abstracts.

Methods: We downloaded 144,149 abstracts from 207 national and international meetings of 63 biomedical conferences. Pairwise comparisons were made using eTBLAST: a text similarity engine. A domain expert then reviewed random samples of highly similar abstracts (1500 total) to estimate the extent of text overlap and possible plagiarism.

Results: Our main findings indicate that the vast majority of textual overlap occurred within the same meeting (2%) and between meetings of the same conference (3%), both of which were significantly higher than instances of plagiarism, which occurred in less than .5% of abstracts.

Conclusions: This analysis indicates that textual overlap in abstracts of papers presented at scientific meetings is one-tenth that of peer-reviewed publications, yet the plagiarism rate is approximately the same as previously measured in peer-reviewed publications. This latter finding underscores a need for monitoring scientific meeting submissions - as is now done when submitting manuscripts to peer-reviewed journals - to improve the integrity of scientific communications.