Empirical evaluation of research prototypes at variable stages of maturity

Abstract

Empirical evaluation of research tools is growing especially in the field of software engineering. A number of research techniques have been proposed and used in evaluating research prototypes. We take the view that evaluation of software engineering tools is best achieved in industrial settings, with real life artifacts and tasks, and with professional software engineers. However, the feasibility of such evaluation is limited for many reasons. Some challenges are related to the prototypes under study, others are related to the industrial environments where the need to meet business requirements take precedence on experimenting with new tools and techniques. In this paper, we summarize our experiences in evaluating a research prototype tool using a grounded theory study, a questionnaire, and a controlled experiment. We discuss the challenges that hindered our industrial evaluation and share ideas on how to overcome these challenges. We propose an action research study where the research tool is used by a small number of experienced professionals in an industrial project.