Tie my hands loosely: Pre-analysis plans in political science.

Abstract

S everal years ago, I was at an American Political Science Association Annual Meeting at which there was a well-attended special session to discuss the newly proposed Data Access and Research Transparency (DART) initiative, which encouraged researchers to make their data available to facilitate evaluation of their findings. During a heated discussion, a very senior scholar who was skeptical of the application of DART in general and for qualitative research in particular, exclaimed in frustration, “It’s as if you think we’re trying to hide things!” A slightly more junior scholar who had been a proponent of greater transparency responded with a deadpan, “Yes, it’s as if you’re trying to hide things.” Much has happened in the 10 or so years since that meeting in terms of social and political science research practice and attitudes. One of the major developments has been the increasingly widespread adoption of the registration of pre-analysis plans (PAPs), in which researchers register their design and empirical specifications before accessing and analyzing (and often before collecting) their data. In many ways, this has been a natural extension of what economists Joshua Angrist and Jorn-Steffen Pischke (2010) call the “credibility revolution” in empirical economics and political economy—broadly speaking, the use of identification-driven research designs, most prominently randomized experiments. Figure 1 shows the growth in the number of registered PAPs in two of the more prominent social science registries—the Evidence in Governance and Politics (EGAP) design registry1 and the American Economic Association’s RCT Registry.2 As can be seen, the number of registrations of PAPs has grown steadily each year since the introduction of these two registries in 2011 and 2013, respectively, with a slight dip (likely pandemic related) in the EGAP registry in 2020. As I alluded to earlier, this trend has been driven to a large extent by the explosion of experimental research designs in political science and economics over the past two decades and by related organizations such as the EGAP research network, which has done much for the adoption of more transparent research practices. There have been several arguments presented for the adoption of PAPs and proposals for how this might work in social science. Humphreys et al. (2013), Nosek et al. (2015), and Munafo et al. (2017) are some of the more prominent recent ones. In this short article, I want to provide some of my thoughts on these developments from the perspective of someone who writes PAPs and reads them as a reviewer, as well as from the perspective of a journal editor.