Improved semantics and implementation through property-based testing with QuickCheck

Abstract

Testing is the primary method to validate that a software implementation meets its specification. In this paper, we demonstrate an approach to validating an executable semantics using property- and model-based random testing in QuickCheck to automate and unify the testing of the semantics and its implementation. Our approach shows the use of executable semantics to bridge the gap between formal mathematical specification and implementation, as well as emphasising the suitability of functional programming languages -- in this case Erlang -- for writing executable semantics. The approach is illustrated through a concrete example, in which the implementation of a proposed extension to the Erlang programming language -- scalable groups -- is tested. This new component comes with a small-step operational semantics written in mathematical notation, and was initially tested using unit testing. Through our work, we were able to find new bugs in both the implementation and the specification.