Conducting experiments on software evolution

Abstract

Software evolution is a challenging issue. For example, how do we judge whether one kind of structure supports evolution better than another kind of structure without comparing them in a fairly controlled way? In this paper we claim that certain aspects of software evolution should be evaluated in controlled experiments with first students, and then professionals, as subjects. Although we may have to compromise on longevity (due to enormous costs), we may observe cause-effect relationships in experiments that cannot be identified in case studies. We describe various challenges of conducting such experiments. We illustrate these issues by a controlled experiment on changeability decay conducted with students and a planned, more realistic extension with mostly software professionals as subjects.