Evaluation of a clinical decision support system for detection of patients at risk after kidney transplantation

Patient care after kidney transplantation requires integration of complex information to make informed decisions on risk constellations. Many machine learning models have been developed for detecting patient outcomes in the past years. However, performance metrics alone do not determine practical utility. Often, the actual performance of medical professionals on the given task is not known. We present a newly developed clinical decision support system (CDSS) for detection of patients at risk for rejection and death-censored graft failure. The CDSS is based on clinical routine data including 1516 kidney transplant recipients and more than 100 000 data points. Additionally, we conduct a reader study to compare the performance of the system to estimations of physicians at a nephrology department with and without the CDSS. Internal validation shows AUC-ROC scores of 0.83 for rejection, and 0.95 for graft failure. The reader study shows that although the predictions by physicians converge towards the suggestions made by the CDSS, performance in terms of AUC-ROC does not improve (0.6413 vs. 0.6314 for rejection; 0.8072 vs. 0.7778 for graft failure). Finally, the study shows that the CDSS detects partially different patients at risk compared to physicians without CDSS. This indicates that the combination of both, medical professionals and a CDSS might help detect more patients at risk for graft failure. However, the question of how to integrate such a system efficiently into clinical practice remains open.