Simulating Fear as Anticipation of Temporal Differences: An experimental investigation

Abstract

Humans use emotional expressions to communicate appraisals. Humans also use emotions in evaluating how they are doing compared to their current goals and desires. The Temporal Difference Reinforcement Learning (TDRL) Theory of Emotion proposes a structure for agents to simulate appropriate emotions during the learning process. In previous work, simulations have shown to reproduce plausible emotion dynamics. In this paper we examine the plausibility and intepretability of TDRL-simulated fear, when expressed by the agent. We presented different TDRL-based fear simulation methods to participants ${\left(n=237\right)}$ in an online study. Each method used a different action selection protocol for the agent's model-based anticipation process. Results suggest that an ${\in}$-greedy fear policy ${\left(\in=0.1\right)}$ combined with a long anticipation horizon provides a plausible fear estimation. This is, to our knowledge, the first experimental evidence detailing some of the predictions of the TDRL Theory of Emotion. Our results are of interest to the design of agent learning methods that are transparent to the user.