When Statistical Significance Is Not Enough: Investigating Relevance, Practical Significance, and Statistical Significance

Abstract

The notions of significance and relevance have provoked much controversy and confusion among those who conduct and those who are intended to be informed by quantitative research in the information systems (IS) field. The history of quantitative research in the IS field and beyond reveals not only disputes over the adequacy of statistical significance to warrant the scientific merits of research, but also pleas for drawing attention to practical significance, as well as a lack of distinction between relevance and practical significance. This essay offers a remedial, overarching account. We establish the position that statistical significance, practical significance, and relevance are distinct qualities, where the latter two transcend mere statistical concerns and respectively refer to the distinct matters of research impressiveness and real-world usefulness. Furthermore, we draw attention to the importance of proper communication of quantitative/statistical analyses through a detailed examination of published IS research. Our examination gives rise to three major issues. The three issues are concerned with the proper communication of (1) research rigor, (2) practical significance, and (3) research relevance. We express our opinions with respect to the three issues and provide a number of recommendations.