External Validity and Meta-Analysis

Abstract

Meta-analysis is a method that combines estimates from studies conducted on different samples, in different contexts, or at different times. Social scientists increasingly use meta-analyses to aggregate evidence and learn about general substantive phenomena. We develop a framework to examine the theoretical foundations of meta-analysis, with emphasis on clarifying the role of external validity. We identify the conditions under which multiple studies are target-equivalent, meaning they identify the same empirical target. Our main result shows that external validity and harmonization, in comparisons made and how outcomes are measured, are necessary and sufficient for target-equivalence. We examine common formulations of meta-analysis—fixed- and random-effects models—developing the theoretical assumptions that underpin them and providing design-based identification results for these models. We then provide practical guidance based on our framework and results. Our results reveal limits to agnostic approaches to the combination of causal evidence from multiple studies.