What Can Be Concluded from Statistical Significance? Severe Testing as an Appealing Extension to Our Standard Toolkit

Abstract

Assessments of statistical significance are ubiquitous in damage quantification practice. Little, however, can be concluded from them on the magnitude of the true effect: statistical significance (against zero) allows the conclusion that the true effect is not zero, but nothing else; and lack of statistical significance does not allow the conclusion that the true effect is zero. Thus, what can be learned? In this note I describe an extension to significance testing, SEVERE TESTING, which does allow valid conclusions on effect sizes after significance testing. It does so on an epistemically appealing, yet technically familiar (p-value) basis. It also makes a difference: loosely speaking, severe testing shifts the evidential weight from the centre of the confidence interval, as often assumed in prevailing practice, to its lower or upper edges.