BMJ Evidence-Based Medicine最新文献

Objective: To assess custom GPT-4 performance in extracting and evaluating data from medical literature to assist in the systematic review (SR) process.

Design: A proof-of-concept comparative study was conducted to assess the accuracy and precision of custom GPT-4 models against human-performed reviews of randomised controlled trials (RCTs).

Setting: Four custom GPT-4 models were developed, each specialising in one of the following areas: (1) extraction of study characteristics, (2) extraction of outcomes, (3) extraction of bias assessment domains and (4) evaluation of risk of bias using results from the third GPT-4 model. Model outputs were compared against data from four SRs conducted by human authors. The evaluation focused on accuracy in data extraction, precision in replicating outcomes and agreement levels in risk of bias assessments.

Participants: Among four SRs chosen, 43 studies were retrieved for data extraction evaluation. Additionally, 17 RCTs were selected for comparison of risk of bias assessments, where both human comparator SRs and an analogous SR provided assessments for comparison.

Intervention: Custom GPT-4 models were deployed to extract data and evaluate risk of bias from selected studies, and their outputs were compared to those generated by human reviewers.

Main outcome measures: Concordance rates between GPT-4 outputs and human-performed SRs in data extraction, effect size comparability and inter/intra-rater agreement in risk of bias assessments.

Results: When comparing the automatically extracted data to the first table of study characteristics from the published review, GPT-4 showed 88.6% concordance with the original review, with <5% discrepancies due to inaccuracies or omissions. It exceeded human accuracy in 2.5% of instances. Study outcomes were extracted and pooling of results showed comparable effect sizes to comparator SRs. A review of bias assessment using GPT-4 showed fair-moderate but significant intra-rater agreement (ICC=0.518, p<0.001) and inter-rater agreements between human comparator SR (weighted kappa=0.237) and the analogous SR (weighted kappa=0.296). In contrast, there was a poor agreement between the two human-performed SRs (weighted kappa=0.094).

Conclusion: Customized GPT-4 models perform well in extracting precise data from medical literature with potential for utilization in review of bias. While the evaluated tasks are simpler than the broader range of SR methodologies, they provide an important initial assessment of GPT-4's capabilities.

Background: Evaluation of the quality of evidence in systematic reviews (SRs) is essential for assertive decision-making. Although Grading of Recommendations Assessment, Development and Evaluation (GRADE) affords a consolidated approach for rating the level of evidence, its application is complex and time-consuming. Artificial intelligence (AI) can be used to overcome these barriers.

Design: Analytical experimental study.

Objective: The objective is to develop and appraise a proof-of-concept AI-powered tool for the semiautomation of an adaptation of the GRADE classification system to determine levels of evidence in SRs with meta-analyses compiled from randomised clinical trials.

Methods: The URSE-automated system was based on an algorithm created to enhance the objectivity of the GRADE classification. It was developed using the Python language and the React library to create user-friendly interfaces. Evaluation of the URSE-automated system was performed by analysing 115 SRs from the Cochrane Library and comparing the predicted levels of evidence with those generated by human evaluators.

Results: The open-source URSE code is available on GitHub (http://www.github.com/alisson-mfc/urse). The agreement between the URSE-automated GRADE system and human evaluators regarding the quality of evidence was 63.2% with a Cohen's kappa coefficient of 0.44. The metrics of the GRADE domains evaluated included accuracy and F1-scores, which were 0.97 and 0.94 for imprecision (number of participants), 0.73 and 0.7 for risk of bias, 0.9 and 0.9 for I² values (heterogeneity) and 0.98 and 0.99 for quality of methodology (A Measurement Tool to Assess Systematic Reviews), respectively.

Conclusion: The results demonstrate the potential use of AI in assessing the quality of evidence. However, in consideration of the emphasis of the GRADE approach on subjectivity and understanding the context of evidence production, full automation of the classification process is not opportune. Nevertheless, the combination of the URSE-automated system with human evaluation or the integration of this tool into other platforms represents interesting directions for the future.

Objectives: To investigate the reporting, data sharing and spin (using reporting strategies to emphasise the benefit of non-significant results) in acupuncture randomised controlled trials (RCTs).

Design: Cross-sectional meta-epidemiological study.

Data sources: Eligible studies indexed in MEDLINE, Embase, CENTRAL, CBM, CNKI, Wanfang Data and VIP Database between 1 January 2014 and 1 May 2024.

Eligibility criteria: Peer-reviewed acupuncture RCTs used traditional medicine (TM), published in English or Chinese, two parallel arms for humans.

Main outcome measures: We assessed (1) the reporting of acupuncture RCTs by the Consolidated Standards for Reporting Trials (CONSORT) 2010 statement and STandards for Reporting Interventions in Clinical Trials of Acupuncture (STRICTA) checklist; (2) the data sharing level by the International Committee of Medical Journal Editors (ICMJE) data sharing statement; (3) spin frequency and level by the prespecified spin strategies.

Results: This study evaluated 476 eligible studies, of which 166 (34.9%) explored the specific efficacy or safety of acupuncture in the nervous system, 68 (14.3%) in the motor system and 61 (12.8%) in the digestive system. 244 (57.7%) studies used conventional acupuncture, 296 (62.2%) used multicentre study design and 369 (77.5%) were supported by institutional funding. 312 (65.5%) eligible studies were poorly reported. The sufficiently reporting scores of the CONSORT 2010 statement and the STRICTA checklist differed from 0.63% to 97.5%, and 32 (59.3%) items were less than 50%. For the data sharing level of acupuncture RCTs, only 66 (17.2%) studies followed the ICMJE data sharing statement, but 49 (14.5%) need to require authors to obtain data, and only 5 (1.5%) provided data by open access. Spins were identified in 408 (85.7%) studies (average spin frequencies: 2.94). 59 (37.2%) studies with non-significant primary outcomes had spin levels.

Conclusions: This study found that the reporting of acupuncture RCTs was low compliance with the CONSORT 2010 statement, the STRICTA checklist and the ICMJE data sharing statement, and spin appeared frequently. Journal policies on using reporting guidelines, data sharing and equitable consideration of non-significant results might enhance the reporting of acupuncture RCTs.

Trial registration number: This study was registered at the Open Science Framework (OSF): (https://doi.org/10.17605/OSF.IO/2WTE6, and https://doi.org/10.17605/OSF.IO/9XDN4,).

Objective: To investigate the efficacy and safety of open-label placebos (OLP) in premenstrual syndrome (PMS).

Design: Randomised controlled trial.

Setting: Switzerland, 2018-2020.

Participants: 150 women (18-45 years of age) with PMS or premenstrual dysphoric disorder.

Intervention: Random assignment (1:1:1) to treatment as usual (TAU), OLP without treatment rationale (OLP-), or OLP with treatment rationale (OLP+). OLP consisted of two placebo pills per day for 6 weeks.

Main outcome measures: Primary outcomes were PMS symptom intensity and interference between groups across three menstrual cycles (MC1-MC3); adverse events (ie, safety) were measured at weeks 3 and 6 after the start of the intervention. Secondary outcomes were psychological and somatic subscales of PMS symptom intensity, and adherence.

Results: From 2 August 2018 to 3 December 2020, 150 women were randomly allocated to TAU (n=50), OLP- (n=50), and OLP+ (n=50), of whom 145 (96.7%) completed trial participation. Groups differed in symptom intensity (F(4)=4.419, p=0.002, r²=0.16) and interference (F(4)=3.159, p=0.014, r²=0.13) across three MCs. Mean symptom intensity at MC3 was lower for OLP+ compared to TAU (b=-9.97, SE=2.85, t(412)=3.50, p<0.001, d=0.90) and to OLP- (b=-6.10, SE=2.89, t(411)=2.11, p=0.036, d=0.55), but OLP- and TAU did not differ (b=-3.87, SE=2.87, t(411)=1.35, p=0.177, d=0.35). Mean interference at MC3 was lower for OLP+ compared to TAU (b=-1.23, SE=0.54, t(443)=2.30, p=0.022, d=0.55) and to OLP- (b=-1.10, SE=0.54, t(442)=2.02, p=0.044, d=0.48), but OLP- and TAU did not differ (b=-0.14, SE=0.54, t(442)=0.26, p=0.799, d=0.06). Four non-serious adverse events were reported in OLP- (n=1) and OLP+ (n=3). Improvement in psychological and somatic symptom intensity was comparable to primary outcomes. Adherence to the OLP intervention was high (93.18±18.95%), with no difference between groups.

Conclusions: The results of our clinical trial indicate that OLP provided with a treatment rationale is an effective, safe, and acceptable treatment for PMS.

Trial registration: ClinicalTrials.gov NCT03547661 (submitted 2 May 2018).