Intervention differential effects and regression to the mean in studies where sample selection is based on the initial value of the outcome variable: an evaluation of methods illustrated in weight-management studies

Abstract

Background: Intervention differential effects (IDEs) occur where changes in an outcome depend upon the initial values of that outcome. Although methods to identify IDEs are well documented, there remains a lack of understanding about the circumstances under which these methods are robust. One context that has not been explored is the identification of intervention differential effect in studies where sample selection is based on the initial value of the outcome being evaluated. We hypothesise that, in such settings, established methods for detecting IDEs will struggle to discriminate these from regression to the mean. Methods: Using simulated datasets of weight-loss intervention programmes that recruit according to initial body mass index, we explore the reliability of Oldham's method and multilevel modelling (MLM) to detect IDEs. Results: In datasets simulated with no IDE, Oldham's method and MLM yield Type I error rates >90%, confirming that threshold selection/truncation leads to bias due to regression to the mean. Type I error rates return close to 5% for both methods when a control group is introduced. Conclusions: Oldham's method and MLM can robustly detect IDEs in this setting, but only if analyses incorporate a control group for comparison.