An empirical analysis and comparison of random testing techniques

Abstract

Testing with randomly generated test inputs, namely Random Testing, is a strategy that has been applied succefully in a lot of cases. Recently, some new adaptive approaches to the random generation of test cases have been proposed. Whereas there are many comparisons of Random Testing with Partition Testing, a systematic comparison of random testing techniques is still missing. This paper presents an empirical analysis and comparison of all random testing techniques from the field of Adaptive Random Testing (ART). The ART algorithms are compared for effectiveness using the mean F-measure, obtained through simulation and mutation analysis, and the P-measure. An interesting connection between the testing effectiveness measures F-measure and P-measure is described. The spatial distribution of test cases is determined to explain the behavior of the methods and identify possible shortcomings. Besides this, both the theoretical asymptotic runtime and the empirical runtime for each method are given.