Comparison of Student – t, Welch s t, and Mann – Whitney U Tests in Terms of Type I Error Rate and Test Power

Abstract

In this study, we compared the Student's t-test, Welch's t-test, and Mann-Whitney U test, in terms of their type I error rate and statistical power when the assumptions of parametric tests are violated in different situations. Materials used in this study, consisted of random numbers generated using the Numpy library in the Python programming language. All random numbers were generated from a normal distribution with N (0, 1) parameters. Balanced and unbalanced experimental conditions were simulated 50 000 times for each combination. The study revealed that, in comparison to other tests, Welch’s t - test was particularly more conservative in terms of type I error rate. It was discovered that the Student-t test had higher power values than the Mann-Whitney U test, mainly when only a small sample size of observations was used for the analysis. This simulation study indicated that Welch’s t - test is robust for preserving type I error rate when the distribution is normal. Therefore, in practice, the use of Welch t-test is recommended based on the findings of this study. One of the recommendations of this study is that the tests in question should also be evaluated in cases where observations have different distributions.