Major mistakes or errors in the use of trial sequential analysis in systematic reviews or meta-analyses – the METSA systematic review

IF 3.9 3区医学 Q1 HEALTH CARE SCIENCES & SERVICES BMC Medical Research Methodology Pub Date : 2024-09-09 DOI:10.1186/s12874-024-02318-y

Christian Gunge Riberholt, Markus Harboe Olsen, Joachim Birch Milan, Sigurlaug Hanna Hafliðadóttir, Jeppe Houmann Svanholm, Elisabeth Buck Pedersen, Charles Chin Han Lew, Mark Aninakwah Asante, Johanne Pereira Ribeiro, Vibeke Wagner, Buddheera W. M. B. Kumburegama, Zheng-Yii Lee, Julie Perrine Schaug, Christina Madsen, Christian Gluud

{"title":"Major mistakes or errors in the use of trial sequential analysis in systematic reviews or meta-analyses – the METSA systematic review","authors":"Christian Gunge Riberholt, Markus Harboe Olsen, Joachim Birch Milan, Sigurlaug Hanna Hafliðadóttir, Jeppe Houmann Svanholm, Elisabeth Buck Pedersen, Charles Chin Han Lew, Mark Aninakwah Asante, Johanne Pereira Ribeiro, Vibeke Wagner, Buddheera W. M. B. Kumburegama, Zheng-Yii Lee, Julie Perrine Schaug, Christina Madsen, Christian Gluud","doi":"10.1186/s12874-024-02318-y","DOIUrl":null,"url":null,"abstract":"Systematic reviews and data synthesis of randomised clinical trials play a crucial role in clinical practice, research, and health policy. Trial sequential analysis can be used in systematic reviews to control type I and type II errors, but methodological errors including lack of protocols and transparency are cause for concern. We assessed the reporting of trial sequential analysis. We searched Medline and the Cochrane Database of Systematic Reviews from 1 January 2018 to 31 December 2021 for systematic reviews and meta-analysis reports that include a trial sequential analysis. Only studies with at least two randomised clinical trials analysed in a forest plot and a trial sequential analysis were included. Two independent investigators assessed the studies. We evaluated protocolisation, reporting, and interpretation of the analyses, including their effect on any GRADE evaluation of imprecision. We included 270 systematic reviews and 274 meta-analysis reports and extracted data from 624 trial sequential analyses. Only 134/270 (50%) systematic reviews planned the trial sequential analysis in the protocol. For analyses on dichotomous outcomes, the proportion of events in the control group was missing in 181/439 (41%), relative risk reduction in 105/439 (24%), alpha in 30/439 (7%), beta in 128/439 (29%), and heterogeneity in 232/439 (53%). For analyses on continuous outcomes, the minimally relevant difference was missing in 125/185 (68%), variance (or standard deviation) in 144/185 (78%), alpha in 23/185 (12%), beta in 63/185 (34%), and heterogeneity in 105/185 (57%). Graphical illustration of the trial sequential analysis was present in 93% of the analyses, however, the Z-curve was wrongly displayed in 135/624 (22%) and 227/624 (36%) did not include futility boundaries. The overall transparency of all 624 analyses was very poor in 236 (38%) and poor in 173 (28%). The majority of trial sequential analyses are not transparent when preparing or presenting the required parameters, partly due to missing or poorly conducted protocols. This hampers interpretation, reproducibility, and validity. PROSPERO CRD42021273811","PeriodicalId":9114,"journal":{"name":"BMC Medical Research Methodology","volume":null,"pages":null},"PeriodicalIF":3.9000,"publicationDate":"2024-09-09","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"BMC Medical Research Methodology","FirstCategoryId":"3","ListUrlMain":"https://doi.org/10.1186/s12874-024-02318-y","RegionNum":3,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"HEALTH CARE SCIENCES & SERVICES","Score":null,"Total":0}

引用次数: 0

Abstract

Systematic reviews and data synthesis of randomised clinical trials play a crucial role in clinical practice, research, and health policy. Trial sequential analysis can be used in systematic reviews to control type I and type II errors, but methodological errors including lack of protocols and transparency are cause for concern. We assessed the reporting of trial sequential analysis. We searched Medline and the Cochrane Database of Systematic Reviews from 1 January 2018 to 31 December 2021 for systematic reviews and meta-analysis reports that include a trial sequential analysis. Only studies with at least two randomised clinical trials analysed in a forest plot and a trial sequential analysis were included. Two independent investigators assessed the studies. We evaluated protocolisation, reporting, and interpretation of the analyses, including their effect on any GRADE evaluation of imprecision. We included 270 systematic reviews and 274 meta-analysis reports and extracted data from 624 trial sequential analyses. Only 134/270 (50%) systematic reviews planned the trial sequential analysis in the protocol. For analyses on dichotomous outcomes, the proportion of events in the control group was missing in 181/439 (41%), relative risk reduction in 105/439 (24%), alpha in 30/439 (7%), beta in 128/439 (29%), and heterogeneity in 232/439 (53%). For analyses on continuous outcomes, the minimally relevant difference was missing in 125/185 (68%), variance (or standard deviation) in 144/185 (78%), alpha in 23/185 (12%), beta in 63/185 (34%), and heterogeneity in 105/185 (57%). Graphical illustration of the trial sequential analysis was present in 93% of the analyses, however, the Z-curve was wrongly displayed in 135/624 (22%) and 227/624 (36%) did not include futility boundaries. The overall transparency of all 624 analyses was very poor in 236 (38%) and poor in 173 (28%). The majority of trial sequential analyses are not transparent when preparing or presenting the required parameters, partly due to missing or poorly conducted protocols. This hampers interpretation, reproducibility, and validity. PROSPERO CRD42021273811

查看原文

微信好友朋友圈 QQ好友复制链接

本刊更多论文

在系统综述或荟萃分析中使用试验序列分析的主要错误或失误 - METSA 系统综述

随机临床试验的系统回顾和数据综合在临床实践、研究和卫生政策中发挥着至关重要的作用。在系统综述中可以使用试验序列分析来控制I型和II型错误，但包括缺乏规程和透明度在内的方法学错误令人担忧。我们对试验序列分析的报告进行了评估。我们检索了2018年1月1日至2021年12月31日期间Medline和Cochrane系统综述数据库中包含试验序列分析的系统综述和荟萃分析报告。只有在森林图和试验序列分析中分析了至少两项随机临床试验的研究才被纳入。两名独立研究人员对研究进行了评估。我们对分析的方案制定、报告和解释进行了评估，包括其对任何 GRADE 不精确性评估的影响。我们纳入了 270 篇系统综述和 274 篇荟萃分析报告，并从 624 项试验序列分析中提取了数据。只有 134 项/270 项（50%）系统综述在方案中规划了试验序列分析。在二分法结果分析中，181/439（41%）的对照组事件比例缺失，105/439（24%）的相对风险降低率缺失，30/439（7%）的α缺失，128/439（29%）的β缺失，232/439（53%）的异质性缺失。对于连续性结果的分析，125/185（68%）的最小相关性差异缺失，144/185（78%）的方差（或标准差）缺失，23/185（12%）的α缺失，63/185（34%）的β缺失，105/185（57%）的异质性缺失。93%的分析中都有试验顺序分析的图表说明，但有135/624（22%）的分析中Z曲线显示错误，227/624（36%）的分析中未包括无效性界限。所有 624 项分析中，236 项（38%）的总体透明度很差，173 项（28%）的透明度很差。大多数试验序列分析在准备或展示所需参数时透明度不高，部分原因是方案缺失或执行不力。这妨碍了解释、可重复性和有效性。prospero crd42021273811

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文去求助

来源期刊

BMC Medical Research Methodology 医学-卫生保健

CiteScore

6.50

自引率

2.50%

发文量

298

审稿时长

3-8 weeks

期刊介绍： BMC Medical Research Methodology is an open access journal publishing original peer-reviewed research articles in methodological approaches to healthcare research. Articles on the methodology of epidemiological research, clinical trials and meta-analysis/systematic review are particularly encouraged, as are empirical studies of the associations between choice of methodology and study outcomes. BMC Medical Research Methodology does not aim to publish articles describing scientific methods or techniques: these should be directed to the BMC journal covering the relevant biomedical subject area.