Different adaptation error types in affective computing have different effects on user experience: A Wizard-of-Oz study

Abstract

In affective computing, classification algorithms are commonly used to recognize users’ psychological states and adapt the technology's behavior accordingly. The accuracy of such classification and adaptation is never perfect, although previous work has shown that user experience improves as the classification/adaptation accuracy increases. This study demonstrates that, apart from accuracy rate, user experience with affective computing technologies also depends on the type of errors made in classification/adaptation. Participants (N = 97) interacted with a modified Multi-Attribute Task Battery scenario whose difficulty level was adapted every 60 s based on participant preferences. Each participant experienced three difficulty adaptation accuracies (70 %, 80 %, 90 %) using a Wizard of Oz paradigm where the “affect-aware” adaptation deviated from the user's preference for a predefined percentage of the time. Participants were randomized into five groups corresponding to different adaptation error types (ranging from “small temporary difficulty change in a nonpreferred direction” to “immediate irreversible increase to overwhelming difficulty”). Self-report measures of user experience included the NASA Task Load Index, Intrinsic Motivation Inventory, and an ad-hoc relative liking scale. Results indicated significant effects of adaptation accuracy on multiple self-report outcomes as well as interaction effects between accuracy and error type on nearly all self-report outcomes. Participants disliked lower accuracies more if they experienced severe adaptation errors. User experience was also different when adaptation errors led to very low difficulty than when they led to very high difficulty. Finally, user experience questionnaires are not necessarily reliable if users do not have prior experience with affective computing technologies. Overall, the study emphasizes that affective computing researchers should not focus exclusively on classification accuracy rates obtained in offline analysis, and that behavioral studies of users actually interacting with affect-aware technologies are needed to fully understand the effects of different technology elements.