Comparative assessment of rating prediction techniques under response uncertainty

Abstract

An objective assessment of collaborative filtering techniques and recommender systems requires application of suitable predictive accuracy metrics. In real life, individuals meet their decisions with considerable uncertainty. This raises the question to what extent the comparison between observed and predicted user responses can be seen as an evident proof of systematic quality differences. In this paper, we accordingly justify underlying assumptions of quality assessment, introduce an appropriate uncertainty-aware evaluation strategy for recommender comparisons, and demonstrate its feasibility and consistency in experiments with real users.