Bayesian and frequentist testing for differences between two groups with parametric and nonparametric two‐sample tests

Abstract

Testing for differences between two groups is one of the scenarios most often faced by scientists across all domains and is particularly important in the medical sciences and psychology. The traditional solution to this problem is rooted inside the Neyman–Pearson theory of null hypothesis significance testing and uniformly most powerful tests. In the last decade, a lot of progress has been made in developing Bayesian versions of the most common parametric and nonparametric two‐sample tests, including Student's t‐test and the Mann–Whitney U test. In this article, we review the underlying assumptions, models and implications for research practice of these Bayesian two‐sample tests and contrast them with the existing frequentist solutions. Also, we show that in general Bayesian and frequentist two‐sample tests have different behavior regarding the type I and II error control, which needs to be carefully balanced in practical research.