Design and rationale of a quality assurance process for a scientific framework

The testing of scientific frameworks is a challenging task. The special characteristics of scientific software e.g. missing test oracle, the need for high performance parallel computing, and high priority of non-functional requirements, need to be accounted for as well as the large variability in a framework. In our previous research, we have shown how software product line engineering can be applied to support the testing of scientific frameworks. We developed a process for handling the variability of a framework using software product line (SPL) variability modeling. From the variability models, we derive test applications and use them for system tests for the framework. In this paper we examine the overall quality assurance for a scientific framework. First, we propose a SPL test strategy for scientific frameworks called Variable test Application strategy for Frameworks (VAF). This test strategy tests both, commonality and variability, of the framework and supports the framework's users in testing their applications by creating reusable test artifacts. We operationalize VAF with test activities that are combined with other quality assurance activities to form the design of a quality assurance process for scientific frameworks. We introduce a list of special characteristics for scientific software that we use as rationale for the design of this process.