Things Go Wrong and the Captain Has to Handle it

How to characterize automation failures? From the point of view of complex, multi-agent operations, I argue that their definition and modeling is the most useful when it accounts for the important drivers of operational performance and safety. This calls us beyond a focus on one-human, one-system performing one task, to a team simultaneously executing many activities in which things are always failing. Much of this activity is defined by the dynamics of the work environment, which can be modeled and predicted. Further, having a human legally responsible for the outcome of the automation’s actions significantly colors the dynamic.