Is open science rewarding A while hoping for B?

Abstract

Reading Guzzo, Schneider, and Nalbantian’s (2022) paper on the unintended consequences of open science policies reminds me of the classic Steven Kerr (1995) paper “On the Folly of Rewarding A, While Hoping for B” in which he makes the point that often we fail to consider the reward systems when we attempt to influence behavior. Such is the case, it seems to me, with the open science movement. Its solution to questionable research practices is to create a system of journal policies that focuses on individual researcher behavior rather than the reward systems that produce it. It is based on an implicit assumption that open science policies prevent questionable research practices and even outright research fraud. Without reforming the reward structure, however, such an outcome is highly unlikely. There are two classes of behavior that open science approaches are designed to reduce: research fraud and questionable research practices. Both arise under a system in which career outcomes are determined by publication in a limited number of exclusive academic outlets. Many departments, particularly in business schools, require publication in a small list of elite “A” journals. Those journals have very narrow requirements that include, as Guzzo et al. note, theory and hypotheses that are confirmed by data. Failure to confirm hypotheses in a paper makes it difficult if not impossible to publish in the places that make one competitive on the academic job market, earn tenure, and reap other rewards (e.g., financial bonuses and reduced teaching loads). Researcher career rewards are based on publishing in the “right” places. Publication rewards (acceptance in top journals) are dependent on not only having theory, hypothesis, and confirmation, but also in convincing reviewers that a given submitted article makes a significant contribution. This means not only filling some perceived gap in the literature but being seen as novel in some way. Finding results that seem counterintuitive can be a good novelty strategy, but they are hard to realize without using a few research tricks. This reward structure—what it takes for career and publication success—puts researchers under tremendous pressure to survive professionally in a cut-throat and hypercompetitive environment where journal acceptance rates in top outlets are about 5%, and authors must fight reviewers over trivial issues through multiple rounds of review. Should we be surprised that so many researchers, facing the dilemma of either gaming the system or finding a different line of work, choose the questionable route? I find myself skeptical that a new set of requirements on researchers will produce a different result.