Operationalizing community-based open scientific design research benchmarks: application to model-based architecture design synthesis

Abstract

The point has repeatedly been made that validation is a crucial success factor in demonstrating the scientific contribution and ensuring the adoption of results. Still, researchers in design science validate their research findings too infrequently. We must all evaluate our claimed contributions on open benchmarks to improve validation quality and foster cumulative research. In this paper, we propose a meta-model to standardise and operationalise the concept of open scientific benchmarks in design science and to guide communities of researchers in the co-development of scientific benchmarks.