Towards automatic functional test execution

Abstract

As applications are developed, functional tests ensure they continue to function as expected. Nowadays, functional testing is mostly done manually, with human testers verifying a system's functionality themselves, following hand-written instructions. While there exist tools supporting functional test automation, in practice they are hard to use, require programming skills, and do not provide good support for test maintenance. In this paper, we take an alternative approach: we semi-automatically convert hand-written instructions into automated tests. Our approach consists of two stages: first, employing machine learning and natural language processing to compute an intermediate representation from test steps; and second, interactively disambiguating that representation to create a fully automated test. These two stages comprise a complete system for converting hand-written functional tests into automated tests. We also present a quantitative study analyzing the effectiveness of our approach. Our results show that 70% of manual test steps can be automatically converted to automated test steps with no user intervention.