Vision: bias in systematic grammar-based test suite construction algorithms

Abstract

The core of grammar-based test suite construction algorithms is a procedure to derive a set of specific phrases, which are then converted into sentences that can be fed into the system under test. This process includes several degrees of freedom and different implementations choose different but ultimately fixed solutions. We show that these fixed choices inherently bias the generated test suite. We quantify these biases and evaluate the effect they have on coverage over the system under test for which the test suite is constructed. We show that the effect of these biases remains prevalent in large real world grammars and systems, even when the test suites grow very large.