Transparent systems, opaque results: a study on automation compliance and task performance.

Abstract

In two experiments, we examine how features of an imperfect automated decision aid influence compliance with the aid in a simplified, simulated nautical collision avoidance task. Experiment 1 examined the impact of providing transparency in the pre-task instructions regarding which attributes of the task that the aid uses to provide its recommendations. Results showed that transparency here positively influenced compliance with the aid, leading to better task performance. Experiment 2 manipulated transparency via confidence estimates presented alongside the aid's recommendations. There were no benefits from this form of transparency. In Experiment 2, lower compliance with the aid's recommendations was found on more difficult collision problems, via a mediating loss of aid reliability and therefore trust. This runs contrary to the hypothesis that harder problems to solve ought to make participants more, rather than less dependent on the aid. Both experiments produced relatively low correlations between trust and compliance. The findings have important implications for the effectiveness of different kinds of transparency implementations, as well as providing a model/framework for understanding how generic factors such as automation reliability and problem difficulty influence both compliance and trust.