How often are replication attempts questioned?

Abstract

Fear of retaliation from the original authors and their allies has been proposed as one of the explanations for the paucity of replications. In the current paper the frequency of negative responses to replications in psychology, and the attention such responses attract, was measured in a series of three studies. Study 1 indicates that replications do not attract more negative mentions in literature than randomly selected non-replication papers unless they are independent and failed, in which case a small increase in negative mentions was noticed, although replications with open data were less likely to attract such mentions. Moreover, no difference in attracting comments on a post-publication peer-review site between replications and non-replication papers was found. Study 2 shows that independent failed and partially successful replications are more likely to attract stand-alone replies than non-replication papers, but the risk is still small and is reduced for replications with open data. Study 3 indicates that stand-alone replies to replications attract fewer citations and readers than the replications to which they respond. I conclude that scientists' unwillingness to criticize published research, cited as one of the reasons for the paucity of replications, also benefits replicators by largely shielding their research from questioning.