Investigating the Adoption of History-based Prioritization in the Context of Manual Testing in a Real Industrial Setting

Abstract

Many test case prioritization techniques have been proposed with the ultimate goal of speeding up fault detection. History-based prioritization, in particular, has been shown to be an effective strategy. Most of the empirical studies conducted on this topic, however, have focused on the context of automated testing. Investigating the effectiveness of history-based prioritization in the context of manual testing is important because, despite the popularity of automated approaches, manual testing is still largely adopted in industry. In this work we propose two history-based prioritization heuristics and evaluate them in the context of manual testing in a real industrial setting. For our evaluation we collected historical test execution information for 23 products, spanning over seven years of historical information, accounting for a total of 2,352 unique test cases and 3,993,863 test results. The results of our experiments showed that the effectiveness of the proposed approach is not far from a theoretical optimal prioritization, and that they are significantly better than alternative orderings of the test suite, including the order suggested by the test management tool and the execution order followed by the testers during the real execution of the test suites evaluated as part of our study.