Good Dog, Bad Dog: Introducing Open Science Badges

Abstract

What does it take to be a good scientist? Most people probably have an implicit idea about good and bad scientific practices. And if you would ask your immediate colleagues at your institute, they would probably easily come up with examples of exemplary scientists. Take, for instance, James, who is an eminent psychologist and who has published extensively on the effects of religion on wellbeing. James always presents his research ideas first during plenary meetings at his department. He is very open about when, where, and how he collects his data. And he sticks to the “four eyes” policy by guaranteeing that at least two people independently check the data and analysis scripts. Most of us would agree that James’s openness and transparency are key scientific virtues that we consider important for making scientific progress. But in a global scientific community and by relying on anonymous peer review, how do we know that our colleagues stick to the rules of the science game? Or how do we know that research projects funded by external agencies, which might have an interest in the outcomes of the study, are not biased toward selectively reporting specific results? Of course, trust is at the basis of all scientific practice. But despite our best efforts, as researchers we all remain prone to a wide variety of different biases that could consciously or subconsciously influence our results (Podsakoff, MacKenzie, Lee, & Podsakoff, 2003). Especially since the replication crisis it is increasingly recognized that many well-known effects in psychology are not as robust as previously thought (Open Science Collaboration, 2015; Simmons, Nelson, & Simonsohn, 2011). Different methodological initiatives have been taken in response to these concerns, such as the launch of multilab replication studies, Bayesian statistics and the use of preregistration (van Elk et al., 2015; van Elk & Wagenmakers, 2017). The Center for Open Science has played a central role in these initiatives, by providing an online platform for preregistering the hypotheses and analysis plans of a study, for sharing the study material and for making the data publicly available. As editors at The International Journal for the Psychology of Religion we believe that these are all welcome developments that provide an important first step in providing more transparency and openness in scientific practice. Therefore, from today on The International Journal for the Psychology of Religion offers the possibility to publish an article with up to three Open Science badges: the Open Data badge, the Open Materials badge, and the Preregistered Badge. These badges have been introduced by the Center for Open Science in response to the replication crisis and the subsequent call for increased transparency and openness in scientific practice. Basically, when submitting an article to The International Journal for the Psychology of Religion as an author you have the option to apply for one or more badges, depending on (a) whether you preregistered the hypotheses and analysis plan for your study, (b) whether the study material is made online available, and (c) whether the data are publicly available. These badges can serve as a “costly signal” to reviewers, colleagues, and readers that will help them to be better able to evaluate and profit from your work (e.g., by using published surveys in a different cultural context; by conducting additional analyses on the data, etc.). Of course a potential danger of the use of “badges” that are indicative of good scientific practices is the judgmental idea of “good dog, bad dog.” Researchers who publish papers with badges can be considered the “best boy of the class,” whereas papers without badges may appear to represent sloppy science. We would like to stress that the use of badges does in no way guarantee the quality and merits of scientific research. On the contrary: Despite all the emphasis on open science and preregistration, we also need to discuss how we can construct good theories and how we can develop