A randomization-based theory for preliminary testing of covariate balance in controlled trials

Abstract

AbstractRandomized trials balance all covariates on average and are the gold standard for estimating treatment effects. Chance imbalances nevertheless exist more or less in realized treatment allocations and intrigue an important question: what should we do if the treatment groups differ with respect to some important baseline characteristics? A common strategy is to conduct a preliminary test of the balance of baseline covariates after randomization, and invoke covariate adjustment for subsequent inference if and only if the realized allocation fails some prespecified criterion. Although such practice is intuitive and popular among practitioners, the existing literature has so far only evaluated its properties under strong parametric model assumptions in theory and simulation, yielding results of limited generality. To fill this gap, we examine two strategies for conducting preliminary test-based covariate adjustment by regression, and evaluate the validity and efficiency of the resulting inferences from the randomization-based perspective. The main result is twofold. First, the preliminary-test estimator based on the analysis of covariance can be even less efficient than the unadjusted difference in means, and risks anticonservative confidence intervals based on normal approximation even with the robust standard error. Second, the preliminary-test estimator based on the fully interacted specification is less efficient than its counterpart under the always-adjust strategy, and yields overconservative confidence intervals based on normal approximation. In addition, although the Fisher randomization test is still finite-sample exact for testing the sharp null hypothesis of no treatment effect on any individual, it is no longer valid for testing the weak null hypothesis of zero average treatment effect in large samples even with properly studentized test statistics. These undesirable properties are due to the asymptotic non-normality of the preliminary-test estimators. Based on theory and simulation, we echo the existing literature and do not recommend the preliminary-test procedure for covariate adjustment in randomized trials.Keywords: Causal inferencedesign-based inferenceefficiencyFisher randomization testregression adjustmentrerandomizationDisclaimerAs a service to authors and researchers we are providing this version of an accepted manuscript (AM). Copyediting, typesetting, and review of the resulting proofs will be undertaken on this manuscript before final publication of the Version of Record (VoR). During production and pre-press, errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal relate to these versions also. FundingThe author(s) reported there is no funding associated with the work featured in this article.