The null need not be nil: Clarifying the parallel arbitrariness of difference testing and equivalence testing

In every statistical analysis, a critical step is to determine the smallest effect size of interest, namely, the arbitrary dividing line between meaningful and negligible results. Different tests address this in different ways, and the contrasting approaches can sometimes lead to confusion. We discuss a key example of such confusion, whereby equivalence testing is perceived to be more arbitrary than difference testing. Our comments are intended to clarify that the latter methods share parallel arbitrariness, and to show how the contrary perception is fueled by the habituated use of “nil null hypotheses” in difference testing. The main premise is that nil null hypotheses give an appearance of objectivity by making the smallest effect size of interest an implicit factor in the interpretation stage of difference testing. When contrasted with the requirements of equivalence testing (where the smallest effect size of interest must be explicitly declared and justified a priori, in the form of the equivalence zone), it is therefore understandable how the misperception of greater arbitrariness could emerge. By combating the latter misperception, our comments serve to promote good practice in both difference testing and equivalence testing.