Critical issues in statistical causal inference for observational physics education research

Abstract

Recent critiques of physics education research (PER) studies have revoiced the critical issues when drawing causal inferences from observational data where no intervention is present. In response to a call for a “causal reasoning primer” in PER, this paper discusses some of the fundamental issues in statistical causal inference. In reviewing these issues, we discuss well-established causal inference methods commonly applied in other fields and discuss their application to PER. Using simulated data sets, we illustrate (i) why analysis for causal inference should control for confounders but not control for mediators and colliders and (ii) that multiple proposed causal models can fit a highly correlated dataset. Finally, we discuss how these causal inference methods can be used to represent and explain existing issues in quantitative PER. Throughout, we discuss a central issue in observational studies: A good quantitative model fit for a proposed causal model is not sufficient to support that proposed model over alternative models. To address this issue, we propose an explicit role for observational studies in PER that draw statistical causal inferences: Proposing future intervention studies and predicting their outcomes. Mirroring the way that theory can motivate experiments in physics, observational studies in PER can predict the causal effects of interventions, and future intervention studies can test those predictions directly.