Deployment Study Length: How Long Should a System Be Evaluated in the Wild?

Abstract

There are many different ways to evaluate a novel interactive system. However, placing that system into the hands of real users and allowing them to use it as they would like in their natural environments may arguably be the best approach to understand if it "really" works. This is because findings learned from user studies conducted in the lab or a controlled setting are limited in external validity and therefore might not generalize beyond the studied usage scenario. Furthermore, the scenario used in a controlled study often lacks full authenticity, and thus it may not faithfully represent situations from the users' live.