On the performance of software testing using multiple versions

Abstract

The authors present analytic models of the performance of comparison checking (also called back-to-back testing and automatic testing), and they use these models to investigate its effectiveness. A Markov model is used to analyze the observation time required for a test system to uncover a fault using comparison checking. A basis for evaluation is provided by developing a similar Markov model for the analysis of ideal checking, i.e. using a perfect (through unrealizable) oracle. Also presented is a model of the effect of comparison checking on a version's failure probability as testing proceeds. Again, comparison checking is evaluated against ideal checking. The analyses show that comparison checking is a powerful and effective technique.<>