Is there a Future for Empirical Software Engineering?

Empirical studies of different kinds are nowadays regularly published in software engineering journals and conferences. Many empirical studies have been published, but are this sufficient? Individual studies are important, but the actual potential in relation to evidence-based software engineering [1] is not fully exploited. As a discipline we have to be able to go further to make our individual studies more useful. Other research should be able to leverage on the studies and industry should be able to make informed decisions based on the empirical research. There are several challenges related to making individual empirical studies useful in a broader context. Anyone having conducted a systematic literature review [2] has most likely experienced the problem of being able to synthesize the relevant studies. In all too many cases, we end up with a systematic mapping study [3], or in the best case something on the borderline between a review and a mapping study. This illustrates the need to write for synthesis [4], and in particular including sufficient contextual information to allow for synthesis [4]. Evidence-based software engineering [1] through the use of systematic literature studies (reviews and maps) has emerged. Methodological support and guidelines (e.g. [2], [3], [6] and [7]) for conducting systematic literature studies have been formulated and they should be carefully followed. However, more is needed! We still need to improve! The keynote is focused on the needs for the future as seen by the presenter. Synthesis has proven hard, and improvements are needed when it comes to both primary studies and secondary studies. It has been shown that the reliability of secondary studies can be challenged [8]. However, if we do manage to publish high quality primary studies, and we truly manage to conduct strong systematic literature reviews, we have a good basis for both building theories in software engineering and to enable industry to make informed decisions using scientific evidence. Unfortunately, this is not the situation today. Theories are mostly based on our own research, as exemplified by [9]. This is fine, but much more can be done if we can easier leverage on the research done by others to build theories. Furthermore, industry is often making decision related to processes, methods, techniques and tools before we manage to obtain sufficient evidence for recommendations. The points made above are highlighted using personal experiences from conducting systematic literature studies, collaborating with industry and research on developing an empirically based software engineering theory.