Linking Theory of Mind in Human-Agent Interactions to Validated Evaluations: Can Explicit Questionnaires Measure Implicit Behaviour?

There is a new crisis emerging in human-agent interaction research: Instead of using validated questionnaires, individual studies are creating new questionnaires that claim to measure identical constructs. This makes replication studies and comparisons between studies near to impossible. In turn, meta-analyses to determine which characteristics are important to create agents that the user experiences as being intelligent are difficult to conduct. As part of the attempt to battle this crisis, in this current paper, we suggest the use of a Theory of Mind task to measure the implicit social behaviour users exhibit towards a virtual agent. In a two-part study, we present findings that suggest that participants conduct this Theory of Mind task as expected: participants adapt towards our virtual agent more than when they conduct the task alone. We additionally present preliminary results correlating performance in the Theory of Mind task to validated constructs. Unfortunately, our current results do not correlate significantly to the existing constructs. Data-collection is ongoing and hence no firm conclusions can be made about this second set of results. However, our data suggest that it is important to become aware that the existing validated constructs used in HCI research may not be tapping into what the researchers assume, and hence provide a basis for important discussions about these implications.