A Reference-Dependent Model of Search Evaluation

Abstract

Most of the existing IR studies employed final values of search behavior measures in building evaluation metrics. However, according to the theories and empirical evidences from Behavioral Economics studies, in people's evaluations of actions and outcomes, the carriers of the values of different actions are gains and losses defined relative to a reference point, rather than the absolute final assets. Based on this idea, I will first explore how users' levels of search satisfaction are affected by the gains and losses defined relative to the pre-search expectations of system performance or reference levels in a controlled lab study. Then, based on the data collected from a field study, I will test the predicative power of my reference-dependent models (built upon delta-value-based behavioral features given the corresponding reference points) in predicting user satisfaction in naturalistic settings, aiming to examine the extent to which the reference-dependent approach can approximate real users' search evaluations. The findings of this work can help us better understand the subjectivity, bias, and variation in users' evaluation of search experience and thus have implications for user modeling and system recommendations design.