Comment on” Estimands and their Estimators for Clinical Trials Impacted by the COVID-19 Pandemic: A Report from the NISS Ingram Olkin Forum Series on Unplanned Clinical Trial Disruptions”

I would like to thank the editor, Prof. Hamasaki, for the opportunity to comment on the thought-provoking work by the NISS working group on unplanned clinical trial disruptions (Van Lancker et al. 2022). The working group’s proposals focus on two basic problems relevant to clinical trials affected by the COVID19 pandemic. The first problem is that, due to the pandemic, the patient population may change systematically over the course of the trial. This raises questions over what is the relevant patient population for which the effect is of interest. The second problem, which receives the major focus in the paper, relates to problems of intercurrent events fueled by the pandemic. The solutions proposed by the working group are interesting and useful. In this commentary, I will nonetheless raise two conceptual shortcomings, which I will attempt to resolve by making more explicit use of methods from causal inference (as opposed to missing data analysis). First, the data collected in a randomized clinical trial are so precious that it is generally difficult to justify ignoring the data collected before or after the start of the pandemic. Those data will often still carry useful information about treatment efficacy, and should ideally be used. Second, whenever possible, analyses of randomized clinical trials should protect the null hypothesis of no treatment effect in the sense that rejection rates should be no larger than the nominal (5%) rate, even when the adopted assumptions fail. Intercurrent events 6 and 7 appear such that they will occur with equal rates in both arms of the trial. If this is so, then this suggests that standard analyses that target the treatment policy estimand, thus ignoring intercurrent events, will protect the null hypothesis of no treatment effect; indeed, the treatment policy estimand then reduces to the balanced estimand of Michiels et al. (2021), which expresses what the treatment effect had been had intercurrent events occurred at “equal rates” in both arms. In this light, analyses that invoke Missing At Random (MAR) assumptions must be taken with caution, as they may be biased whenever the MAR assumption fails. More importantly, analyses that explicitly combine biased and unbiased estimators, as in