Trends in null hypothesis significance testing: Still going strong.

Abstract

Null hypothesis significance testing (NHST) is among the most prominent and widely used methods for analyzing data. At the same time, NHST has been criticized since many years because of misuses and misconceptions that can be found extensively in the scholarly literature. Furthermore, in recent years, NHST has been identified as one reason for the replication crisis because many studies place too much emphasis on statistical significance for drawing conclusions. As a response to those problems, calls for actions have been raised, among others by the American Statistical Association (ASA), to rectify these issues, for instance, by modifying or even abandoning NHST. In this paper, we study the reaction of the community on these discussions. Specifically, we conduct a scientometric analysis of bibliographic records to investigate the publication behavior about the usage of NHST. We conduct a trend analysis for the general community, for specific subject areas and for individual journals. Furthermore, we conduct a change-point analysis to investigate if there are continued movements or actual changes. As a result, we find that for the general community NHST is more popular than ever, however, for particular subject-areas and journals there is a clear heterogeneity and no uniform publication behavior is observable.