A meta-epidemiological analysis of post-hoc comparisons and primary endpoint interpretability among randomized noncomparative trials in clinical medicine

Abstract

Objectives

Randomized noncomparative trials (RNCTs) promise reduced accrual requirements vs randomized controlled comparative trials because RNCTs do not enroll a control group and instead compare outcomes to historical controls or prespecified estimates. We hypothesized that RNCTs often suffer from two methodological concerns: (1) lack of interpretability due to group-specific inferences in nonrandomly selected samples and (2) misinterpretation due to unlicensed between-group comparisons lacking prespecification. The purpose of this study was to characterize RNCTs and the incidence of these two methodological concerns.

Study Design and Setting

We queried PubMed and Web of Science on September 14, 2023, to conduct a meta-epidemiological analysis of published RNCTs in any field of medicine. Trial characteristics and the incidence of methodological concerns were manually recorded.

Results

We identified 70 RNCTs published from 2002 to 2023. RNCTs have been increasingly published over time (slope = 0.28, 95% CI 0.17–0.39, P < .001). Sixty trials (60/70, 86%) had a lack of interpretability for the primary endpoint due to group-specific inferences. Unlicensed between-group comparisons were present in 36 trials (36/70, 51%), including in the primary conclusion of 31 trials (31/70, 44%), and were accompanied by significance testing in 20 trials (20/70, 29%). Only five (5/70, 7%) trials were found to have neither of these flaws.

Conclusion

Although RNCTs are increasingly published over time, the primary analysis of nearly all published RNCTs in the medical literature was unsupported by their fundamental underlying methodological assumptions. RNCTs promise group-specific inference, which they are unable to deliver, and undermine the primary advantage of randomization, which is comparative inference. The ongoing use of the RNCT design in lieu of a traditional randomized controlled comparative trial should therefore be reconsidered.

Plain Language Summary

The typical way that doctors can learn whether new drugs are helpful is through a clinical trial. Often, doctors compare these new treatments to the control treatment being used in standard clinical practice. When researchers want to compare different treatments, they may decide to randomly assign one treatment or the other to trial participants. Like flipping a coin, randomly deciding which treatment to use can help researchers make the best comparisons between the new and control treatment by limiting certain biases. These trials are called “randomized comparative trials” and are the most common way researchers can improve medicine. A newer type of trial, called a “randomized noncomparative trial,” has become increasingly popular in medicine. Like randomized comparative trials, this type of trial randomly decides which treatment participants receive. However, the “randomized noncomparative trial” is not designed to evaluate whether the new treatment results in better outcomes compared with the control treatment. Instead, the results of each randomized arm in the trial are compared to other patients, who are not a part of the trial, or to another measure set ahead of time by the researchers. This is justified by some researchers, who say that fewer participants are needed for such trials, which helps to finish the trial faster. However, directly comparing the results of patients after receiving a treatment on a clinical trial is one of the most important parts of the trial and the main reason why researchers would want to randomly assign treatments in the first place. To better understand how RNCTs are used in practice, we reviewed all such trials that have been completed in medicine to date. We found that more than half of RNCTs actually ended up comparing their patients anyway, despite saying they would not. This is a problem because these comparisons are not prespecified and may therefore be only reported when the result is what the researchers wanted. Furthermore, the main outcome of each trial was difficult to interpret in most trials because there was no effort to show that the enrolled patients were representative of any prespecified population that would facilitate comparisons with historical information. Overall, only five trials, or just 7% of the “randomized noncomparative trials” published in the medical literature, did not have either of these issues. As a result, this type of clinical trial does not seem to be a good way to improve medical care. If researchers want to learn which treatments are better, they should stick to the standard way to learn this—randomized comparative trials.