Bamboozled by Bonferroni

Abstract

When many statistical hypotheses are evaluated simultaneously, statisticians often recommend adjusting (or correcting) standard hypothesis tests. In this paper, I (1) distinguish two senses of adjustment, (2) investigate the prudential and epistemic goals that adjustment might achieve, and (3) identify conditions under which a researcher should not adjust for multiplicity in the two senses I identify. I tentatively conclude that the goals of scientists and the public may be misaligned with the decision criteria used to evaluate multiple testing regimes.