A framework to characterise the reproducibility of meta-analysis results with its application to direct oral anticoagulants in the acute treatment of venous thromboembolism

The number of meta-analyses of aggregate data has dramatically increased due to the facility of obtaining data from publications and the development of free, easy-to-use, and specialised statistical software. Even when meta-analyses include the same studies, their results may vary owing to different methodological choices. Assessment of the replication of meta-analysis provides an example of the variation of effect ‘naturally’ observed between multiple research projects. Reproducibility of results has mostly been reported using graphical descriptive representations. A quantitative analysis of such results would enable (i) breakdown of the total observed variability with quantification of the variability generated by the replication process and (ii) identification of which variables account for this variability, such as methodological quality or the statistical analysis procedures used. These variables might explain systematic mean differences between results and dispersion of the results. To quantitatively characterise the reproducibility of meta-analysis results, a bivariate linear mixed-effects model was developed to simulate both mean results and their corresponding uncertainty. Results were assigned to several replication groups, those assessing the same studies, outcomes, treatment indication and comparisons classified in the same replication group. A nested random effect structure was used to break down the total variability within each replication group and between these groups to enable calculation of an intragroup correlation coefficient and quantification of reproducibility. Determinants of variability were investigated by modelling both mean and variance parameters using covariates. The proposed model was applied to the example of meta-analyses evaluating direct oral anticoagulants in the acute treatment of venous thromboembolism.