Glenn T Werneburg, Eric A Werneburg, Howard B Goldman, Emily Slopnick, Ly Hoang Roberts, Sandip P Vasavada
{"title":"外部验证表明,机器学习模型在预测客观和患者报告的膀胱过度活动症治疗效果方面优于人类专家。","authors":"Glenn T Werneburg, Eric A Werneburg, Howard B Goldman, Emily Slopnick, Ly Hoang Roberts, Sandip P Vasavada","doi":"10.1016/j.urology.2024.08.071","DOIUrl":null,"url":null,"abstract":"<p><strong>Objective: </strong>To predict treatment response for overactive bladder (OAB) for a specific patient remains elusive. We sought to develop accurate models using machine learning for prediction of objective and patient-reported treatment response to intravesical botulinum toxin (OBTX-A) injection. We sought to validate the models in a challenging setting using an external dataset of a markedly different patient cohort and dosing regimen. We hypothesized the model would outperform human experts and top available algorithms.</p><p><strong>Methods: </strong>Algorithms using \"operator splitting\" designed for accuracy and efficiency even in small training datasets with variable completeness, were trained to predict objective response and patient-reported symptomatic improvement using the ROSETTA trial cohort and validated using the ABC trial cohort of patients who underwent OBTX-A. Areas under the curve (AUC) of algorithms were compared to the top publicly-available machine-learning classifier XGBoost, logistic regression with cross validation, and human expert predictions in the external validation set.</p><p><strong>Results: </strong>In the validation set, the operator splitting neural network had AUC of 0.66 and outperformed XGBoost with DART (top available machine-learning classifier, AUC: 0.58), logistic regression (AUC 0.55), and human experts (AUC 0.47-0.53) for prediction of clinical responder status. It was similarly accurate in prediction of patient subjective improvement in symptoms following OBTX-A (AUC: 0.64), again outperforming other algorithms and human experts (AUC 0.41-0.62).</p><p><strong>Conclusion: </strong>The neural network outperformed human experts and other machine-learning approaches in prediction of objective and patient-reported OBTX-A outcomes for OAB in a challenging independent validation cohort. Clinical implementation could improve counseling and treatment selection.</p>","PeriodicalId":23415,"journal":{"name":"Urology","volume":null,"pages":null},"PeriodicalIF":2.1000,"publicationDate":"2024-09-04","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":"{\"title\":\"External Validation Demonstrates Machine Learning Models Outperform Human Experts in Prediction of Objective and Patient-Reported Overactive Bladder Treatment Outcomes.\",\"authors\":\"Glenn T Werneburg, Eric A Werneburg, Howard B Goldman, Emily Slopnick, Ly Hoang Roberts, Sandip P Vasavada\",\"doi\":\"10.1016/j.urology.2024.08.071\",\"DOIUrl\":null,\"url\":null,\"abstract\":\"<p><strong>Objective: </strong>To predict treatment response for overactive bladder (OAB) for a specific patient remains elusive. We sought to develop accurate models using machine learning for prediction of objective and patient-reported treatment response to intravesical botulinum toxin (OBTX-A) injection. We sought to validate the models in a challenging setting using an external dataset of a markedly different patient cohort and dosing regimen. We hypothesized the model would outperform human experts and top available algorithms.</p><p><strong>Methods: </strong>Algorithms using \\\"operator splitting\\\" designed for accuracy and efficiency even in small training datasets with variable completeness, were trained to predict objective response and patient-reported symptomatic improvement using the ROSETTA trial cohort and validated using the ABC trial cohort of patients who underwent OBTX-A. Areas under the curve (AUC) of algorithms were compared to the top publicly-available machine-learning classifier XGBoost, logistic regression with cross validation, and human expert predictions in the external validation set.</p><p><strong>Results: </strong>In the validation set, the operator splitting neural network had AUC of 0.66 and outperformed XGBoost with DART (top available machine-learning classifier, AUC: 0.58), logistic regression (AUC 0.55), and human experts (AUC 0.47-0.53) for prediction of clinical responder status. It was similarly accurate in prediction of patient subjective improvement in symptoms following OBTX-A (AUC: 0.64), again outperforming other algorithms and human experts (AUC 0.41-0.62).</p><p><strong>Conclusion: </strong>The neural network outperformed human experts and other machine-learning approaches in prediction of objective and patient-reported OBTX-A outcomes for OAB in a challenging independent validation cohort. Clinical implementation could improve counseling and treatment selection.</p>\",\"PeriodicalId\":23415,\"journal\":{\"name\":\"Urology\",\"volume\":null,\"pages\":null},\"PeriodicalIF\":2.1000,\"publicationDate\":\"2024-09-04\",\"publicationTypes\":\"Journal Article\",\"fieldsOfStudy\":null,\"isOpenAccess\":false,\"openAccessPdf\":\"\",\"citationCount\":\"0\",\"resultStr\":null,\"platform\":\"Semanticscholar\",\"paperid\":null,\"PeriodicalName\":\"Urology\",\"FirstCategoryId\":\"3\",\"ListUrlMain\":\"https://doi.org/10.1016/j.urology.2024.08.071\",\"RegionNum\":3,\"RegionCategory\":\"医学\",\"ArticlePicture\":[],\"TitleCN\":null,\"AbstractTextCN\":null,\"PMCID\":null,\"EPubDate\":\"\",\"PubModel\":\"\",\"JCR\":\"Q2\",\"JCRName\":\"UROLOGY & NEPHROLOGY\",\"Score\":null,\"Total\":0}","platform":"Semanticscholar","paperid":null,"PeriodicalName":"Urology","FirstCategoryId":"3","ListUrlMain":"https://doi.org/10.1016/j.urology.2024.08.071","RegionNum":3,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q2","JCRName":"UROLOGY & NEPHROLOGY","Score":null,"Total":0}
External Validation Demonstrates Machine Learning Models Outperform Human Experts in Prediction of Objective and Patient-Reported Overactive Bladder Treatment Outcomes.
Objective: To predict treatment response for overactive bladder (OAB) for a specific patient remains elusive. We sought to develop accurate models using machine learning for prediction of objective and patient-reported treatment response to intravesical botulinum toxin (OBTX-A) injection. We sought to validate the models in a challenging setting using an external dataset of a markedly different patient cohort and dosing regimen. We hypothesized the model would outperform human experts and top available algorithms.
Methods: Algorithms using "operator splitting" designed for accuracy and efficiency even in small training datasets with variable completeness, were trained to predict objective response and patient-reported symptomatic improvement using the ROSETTA trial cohort and validated using the ABC trial cohort of patients who underwent OBTX-A. Areas under the curve (AUC) of algorithms were compared to the top publicly-available machine-learning classifier XGBoost, logistic regression with cross validation, and human expert predictions in the external validation set.
Results: In the validation set, the operator splitting neural network had AUC of 0.66 and outperformed XGBoost with DART (top available machine-learning classifier, AUC: 0.58), logistic regression (AUC 0.55), and human experts (AUC 0.47-0.53) for prediction of clinical responder status. It was similarly accurate in prediction of patient subjective improvement in symptoms following OBTX-A (AUC: 0.64), again outperforming other algorithms and human experts (AUC 0.41-0.62).
Conclusion: The neural network outperformed human experts and other machine-learning approaches in prediction of objective and patient-reported OBTX-A outcomes for OAB in a challenging independent validation cohort. Clinical implementation could improve counseling and treatment selection.
期刊介绍:
Urology is a monthly, peer–reviewed journal primarily for urologists, residents, interns, nephrologists, and other specialists interested in urology
The mission of Urology®, the "Gold Journal," is to provide practical, timely, and relevant clinical and basic science information to physicians and researchers practicing the art of urology worldwide. Urology® publishes original articles relating to adult and pediatric clinical urology as well as to clinical and basic science research. Topics in Urology® include pediatrics, surgical oncology, radiology, pathology, erectile dysfunction, infertility, incontinence, transplantation, endourology, andrology, female urology, reconstructive surgery, and medical oncology, as well as relevant basic science issues. Special features include rapid communication of important timely issues, surgeon''s workshops, interesting case reports, surgical techniques, clinical and basic science review articles, guest editorials, letters to the editor, book reviews, and historical articles in urology.