Observational studies最新文献

Judea Pearl, quoted in Pearl and Mackenzie (2008), stated that "once we have understood why [randomized controlled trials] RCTs work, there is no need to put them on a pedestal and treat them as the gold standard of causal analysis, which all other methods should emulate." In Aronow et al. (2024), this claim is refuted, drawing on results of Robins and Ritov (1997). The argument is made that statistical estimation and inference tend to be fundamentally more difficult in observational studies than in randomized controlled trials, even when all confounders are observed and measured without error. We congratulate the authors for raising this highly timely, interesting discussion and welcome this opportunity to join this important debate. In this commentary, we focus on what it takes to generate reliable evidence across a spectrum of studies that increasingly involve real-world elements and less control over design. A related question is whether, along this spectrum of studies, the reliability of evidence generated by a statistical analysis decreases. We claim that this is not the case, but that the challenge for the appropriate statistical method increases, requiring sophisticated and careful execution.

We argue that randomized controlled trials (RCTs) are special even among studies for which a nonparametric unconfoundedness assumption is credible. This claim follows from two results of Robins and Ritov (1997). First, in settings with at least one continuous confounder, there exists no estimator of the average treatment effect that is uniformly consistent unless the propensity score is known or additional assumptions are made on the complexity of the propensity score function. Second, with binary outcomes, knowledge of the propensity score yields a uniformly consistent estimator and finite-sample valid confidence intervals that shrink at a parametric rate, regardless of how complicated the propensity score function might be. We emphasize the latter point, and note that a successfully executed RCT provides knowledge of the propensity score to the researcher. We conclude that statistical estimation and inference tend to be fundamentally more difficult in observational settings than in RCTs, even when all confounders are observed and measured without error.

In response to Pearl, Aronow et al. (2025) argue that randomized experiments are special among causal inference methods due to their statistical properties. I believe that the key distinction between randomized experiments and observational studies is not statistical, but rather epistemological in nature. In this comment, I aim to articulate this epistemological distinction and argue that it ought to take a more central role in these discussions.

We provide a critical response to Aronow et al. (2021) which argued that randomized controlled trials (RCTs) are "enough," while nonparametric identification in observational studies is not. We first investigate what is meant by "enough," arguing that this is a fundamentally a sociological claim about the relationship between statistical work and relevant institutional processes (here, academic peer review), rather than something that can be decided from within the logic of statistics. For a more complete conception of "enough," we outline all that would need to be known - not just knowledge of propensity scores, but knowledge of many other spatial and temporal characteristics of the social world. Even granting the logic of the critique in Aronow et al. (2021), its practical importance is a question of the contexts under study. We argue that we should not be satisfied by appeals to intuition or experience about the complexity of "naturally occurring" propensity score functions. Instead, we call for more empirical metascience to begin to characterize this complexity. We apply this logic to the case of recommender systems as a demonstration of the weakness of allowing statisticians' intuitions to serve in place of metascientific data. This may be, as Aronow et al. (2021) claim, one of the "few free lunches in statistics"-but like many of the free lunches consumed by statisticians, it is only available to those working at a handful of large tech firms. Rather than implicitly deciding what is "enough" based on statistical applications the social world has determined to be most profitable, we are argue that practicing statisticians should explicitly engage with questions like "for what?" and "for whom?" in order to adequately answer the question of "enough?"

We thank the editor for organizing a diverse and wide-ranging discussion, and we thank the commentators for their detailed and thoughtful remarks. Most of the commentators provide broader perspectives on randomized experiments and their role in modern empirical practice. We believe this broader perspective is important, and the comments serve as complements to the somewhat narrow points we made in our paper. However, we believe these narrow points are of great consequence, and we find it useful to briefly recapitulate them here. When a practitioner aims to estimate averages of bounded potential outcomes (e.g., the average treatment effect on a binary outcome) in a setting where both ignorability and positivity are known to hold after adjusting for at least one continuous covariate, the following statements are true: • If the propensity score is known, such as in a randomized controlled trial (RCT), there exist simple estimators that are uniformly root-n consistent and asymptotically normal. Confidence intervals based on these estimators are finite-sample valid and their widths shrink at a root-n rate. • If the propensity score is not known, such as in an observational study, there exist neither uniformly consistent estimators nor uniform (i.e., honest) large-sample confidence intervals whose widths are shrinking with the sample size. To achieve these properties, the practitioner must impose untestable assumptions on either the propensity score function or the conditional expectation function of the outcomes.

This commentary proposes a framework for understanding the role of statistics in policymaking, regulation, and bureaucratic systems. I introduce the concept of "ex ante policy," describing statistical rules and procedures designed before data collection to govern future actions. Through examining examples, particularly clinical trials, I explore how ex ante policy serves as a calculus of bureaucracy, providing numerical foundations for governance through clear, transparent rules. The ex ante frame obviates heated debates about inferential interpretations of probability and statistical tests, p-values, and rituals. I conclude by calling for a deeper appreciation of statistics' bureaucratic function and suggesting new directions for research in policy-oriented statistical methodology.

Aronow et al. (2025) provide a convincing case for the special status of randomized controlled trials (RCTs) in which the propensity scores are known and can be used to make causal inferences. Here we provide a Bayesian perspective on their work by summarizing recent developments in the Bayesian literature on the topic. Whether the propensity score should play a role in Bayesian causal inference - and what that role(s) should be - has been a controversial topic for some time. We begin by describing Bayesian inference for population-level estimands and show that under commonly made (but not necessarily required) assumptions, the propensity score model has no role to play in Bayesian causal inference from a purist perspective. We discuss recent work on why these assumptions can be problematic - particularly in high-dimensional models - and discuss several Bayesian motivations for relaxing them. We describe out recent approaches for incorporating the propensity score correspond to di erent ways of relaxing these assumptions. Given these considerations, we illustrate how a Bayesian might approach the synethic examples of Aronow et al. (2025).

Aronow et al. (2024) provide a great service to the causal inference community by delineating the key results in Robins and Ritov (1997). They show that randomized controlled trials (RCTs) ensure much stronger statistical inference than unconfounded observational studies even though nonparametric identification is identical in both settings. These results are in sharp contrast to the claim in Pearl and Mackenzie (2018) that RCTs are not the gold standard of causal analysis. Pearl and Mackenzie's (2018) claim is false and misleading for empirical researchers who want to infer causal effects based on data with finite sample sizes. I will further review what randomization can and cannot guarantee more broadly. In particular, I will highlight the value of randomization-based inference in RCTs, the limit of randomization alone for more complicated causal inference questions, and the importance of sensitivity analysis in observational studies.