On the ecological validity of a password study

Abstract

The ecological validity of password studies is a complex topic and difficult to quantify. Most researchers who conduct password user studies try to address the issue in their study design. However, the methods researchers use to try to improve ecological validity vary and some methods even contradict each other. One reason for this is that the very nature of the problem of ecological validity of password studies is hard to study, due to the lack of ground truth. In this paper, we present a study on the ecological validity of password studies designed specifically to shed light on this issue. We were able to compare the behavior of 645 study participants with their real world password choices. We conducted both online and laboratory studies, under priming and non-priming conditions, to be able to evaluate the effects of these different forms of password studies. While our study is able to investigate only one specific password environment used by a limited population and thus cannot answer all questions about ecological validity, it does represent a first important step in judging the impact of ecological validity on password studies.