A Question of Trust: Old and New Metrics for the Reliable Assessment of Trustworthy AI

: This work contributes to the evaluation of the quality of decision support systems constructed with Machine Learning (ML) techniques in Medical Artificial Intelligence (MAI). In particular, we propose and discuss metrics that complement and go beyond traditional assessment practices based on the evaluation of accuracy, by focusing on two different dimensions related to the trustworthiness of a MAI system: reputation/ability, which relates to the accuracy or predictive ability of the system itself; and expertise/source reliability, which relates instead to the trustworthiness of the data which have been used to construct the MAI system. Then, we will discuss some previous, but so far mostly neglected, proposals as well novel metrics, visualizations and procedures for the sound evaluation of a MAI system’s trustworthiness, by focusing on six different concepts: advice accuracy, advice reliability, pragmatic utility, advice value, decision benefit and potential robustness. Finally, we will illustrate the application of the proposed concepts through two realistic medical case studies.