Examining uncertainty in journal peer reviewers' recommendations: a cross-sectional study.

Abstract

The peer review process is used throughout science but has often been criticized for being inconsistent, with decisions dependent on the peers who did the reviewing. Much of the decision inconsistency arises from the differences between reviewers in terms of their expertise, training and experience. Another source of uncertainty is within reviewers as they must make a single recommendation (e.g. 'Accept'), when they may have wavered between two (e.g. 'Accept' or 'Reject'). We estimated the size of within-reviewer uncertainty using post-review surveys at three journals. We asked reviewers to think outside the recommendation they gave (e.g. 'Accept') and assign percentages to all other recommendations (e.g. 'Major revision'). Reviewers who were certain could assign 100% to one recommendation. Twenty-three per cent of reviewers reported no uncertainty (95% confidence interval 19-27%). Women were associated with more uncertainty at one journal, and protocol papers were associated with more uncertainty at one journal. Reviewers commonly experience some uncertainty when peer-reviewing journal articles. This uncertainty is part of the variability in peer reviewers' recommendation.