Risk-Based Testing of Self-Adaptive Systems Using Run-Time Predictions

Abstract

Devising test strategies for specific test goals relies on predictions of the run-time behavior of the software system under test (SuT) based on specifications, models, or the code. For a system following a single strategy as run-time behavior, the test strategy can be fixed at design time. For an adaptive system, which may choose from several strategies due to environment changes, a combination of test strategies has to be found, which still can be achieved at design time provided that all system strategies and the switching policy are predictable. Self-adaptive systems, also adapting their system strategies and strategy switches according to the environmental dynamics, render such design-time predictions futile, but there also the test strategies have to be adapted at run time. We characterize the necessary interplay between system strategy adaptation of the SuT and test strategy adaptation of the tester as a Stochastic Game. We argue that the tester's part, formalized by means of a Markov Decision Process, can be automatically solved by the use of Reinforcement Learning methods where we discuss both model-based and model-free variants. Finally, we propose a particular framework inspired by Direct Future Prediction which, given a simulation of the SuT and its environment, autonomously finds good test strategies w.r.t. imposed quanti?able goals. While these goals, in general, can be initialized arbitrarily, our evaluation concentrates on risk-based goals rewarding the detection of hazardous failures.