XAI in hindsight: Shapley values for explaining prediction accuracy

Predicting the outcome of AI-models is inherently difficult, and understanding and trusting the models and decisions based on them are challenging. To help us, various explainable artificial intelligence (XAI) methods have been developed. Sometimes, explanations are requested in hindsight, for example after an accident has occurred due to unexpected or erroneous model outcomes. In such situations, our focus is on answering why the model failed to produce an accurate prediction. But since XAI-methods are typically made without knowledge of the true outcome values, the explanations concern the prediction and not the prediction error. In this paper, we change perspective and assume that the true values are known and propose an explanation method that quantifies how different subsets/clusters of the training data impact how the predicted values deviate from the true values, the so-called residuals. In this way, the proposed method lets us explain the accuracy of individual predictions, in hindsight. By focusing on explanations in hindsight, rather than the predictions per se, the proposed method offers a novel perspective to the field of XAI. The method is demonstrated and evaluated using both synthetic and real-world data. To objectively evaluate the method we propose, we utilize the explanations we generate to tailor a training data acquisition strategy and show how this leads to improved prediction performance. The proposed method is fully generic, and applicable to any industry. In the presentation offered here, most examples are related to the maritime industry.