Schwarzwälder — An online test framework

Abstract

This paper describes a testing and benchmarking framework built to support the community of computer scientists. The framework provides features for defining benchmarks, uploading program code and executing test runs on the test infrastructure. For every test run, the framework measures the performance of the execution in terms of execution time, resource consumption and the quality of the data output computed by the tested code. As tests may be defined for many kinds of algorithmic problems the framework could support the scientific community working on Digital Ecosystem Technologies by achieving comparability of multiple algorithmic approaches to existing problems. The community may define its own tests on the framework thereby starting a worldwide online competition for the best algorithms.