Prediction Under Interventions: Evaluation of Counterfactual Performance Using Longitudinal Observational Data.

IF 4.7 2区 医学 Q1 PUBLIC, ENVIRONMENTAL & OCCUPATIONAL HEALTH Epidemiology Pub Date : 2024-04-18 DOI:10.1097/ede.0000000000001713
Ruth H Keogh, Nan Van Geloven
{"title":"Prediction Under Interventions: Evaluation of Counterfactual Performance Using Longitudinal Observational Data.","authors":"Ruth H Keogh, Nan Van Geloven","doi":"10.1097/ede.0000000000001713","DOIUrl":null,"url":null,"abstract":"Predictions under interventions are estimates of what a person's risk of an outcome would be if they were to follow a particular treatment strategy, given their individual characteristics. Such predictions can give important input to medical decision-making. However, evaluating the predictive performance of interventional predictions is challenging. Standard ways of evaluating predictive performance do not apply when using observational data, because prediction under interventions involves obtaining predictions of the outcome under conditions that are different from those that are observed for a subset of individuals in the validation dataset. This work describes methods for evaluating counterfactual performance of predictions under interventions for time-to-event outcomes. This means we aim to assess how well predictions would match the validation data if all individuals had followed the treatment strategy under which predictions are made. We focus on counterfactual performance evaluation using longitudinal observational data, and under treatment strategies that involve sustaining a particular treatment regime over time. We introduce an estimation approach using artificial censoring and inverse probability weighting that involves creating a validation dataset mimicking the treatment strategy under which predictions are made. We extend measures of calibration, discrimination (c-index and cumulative/dynamic AUCt) and overall prediction error (Brier score) to allow assessment of counterfactual performance. The methods are evaluated using a simulation study, including scenarios in which the methods should detect poor performance. Applying our methods in the context of liver transplantation shows that our procedure allows quantification of the performance of predictions supporting crucial decisions on organ allocation.","PeriodicalId":11779,"journal":{"name":"Epidemiology","volume":"51 1","pages":""},"PeriodicalIF":4.7000,"publicationDate":"2024-04-18","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Epidemiology","FirstCategoryId":"3","ListUrlMain":"https://doi.org/10.1097/ede.0000000000001713","RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"PUBLIC, ENVIRONMENTAL & OCCUPATIONAL HEALTH","Score":null,"Total":0}
引用次数: 0

Abstract

Predictions under interventions are estimates of what a person's risk of an outcome would be if they were to follow a particular treatment strategy, given their individual characteristics. Such predictions can give important input to medical decision-making. However, evaluating the predictive performance of interventional predictions is challenging. Standard ways of evaluating predictive performance do not apply when using observational data, because prediction under interventions involves obtaining predictions of the outcome under conditions that are different from those that are observed for a subset of individuals in the validation dataset. This work describes methods for evaluating counterfactual performance of predictions under interventions for time-to-event outcomes. This means we aim to assess how well predictions would match the validation data if all individuals had followed the treatment strategy under which predictions are made. We focus on counterfactual performance evaluation using longitudinal observational data, and under treatment strategies that involve sustaining a particular treatment regime over time. We introduce an estimation approach using artificial censoring and inverse probability weighting that involves creating a validation dataset mimicking the treatment strategy under which predictions are made. We extend measures of calibration, discrimination (c-index and cumulative/dynamic AUCt) and overall prediction error (Brier score) to allow assessment of counterfactual performance. The methods are evaluated using a simulation study, including scenarios in which the methods should detect poor performance. Applying our methods in the context of liver transplantation shows that our procedure allows quantification of the performance of predictions supporting crucial decisions on organ allocation.
查看原文
分享 分享
微信好友 朋友圈 QQ好友 复制链接
本刊更多论文
干预下的预测:利用纵向观察数据评估反事实绩效。
干预措施下的预测是指根据一个人的个体特征,估计如果他采取某种治疗策略,会有多大的结果风险。此类预测可为医疗决策提供重要参考。然而,评估干预预测的预测性能是一项挑战。评估预测性能的标准方法不适用于使用观察数据的情况,因为干预预测涉及在不同于对验证数据集中的子集个体进行观察的条件下获得结果预测。这项工作介绍了评估时间到事件结果干预下预测的反事实性能的方法。这意味着,我们的目标是评估,如果所有个体都采用了预测所依据的治疗策略,预测结果与验证数据的匹配程度如何。我们的重点是利用纵向观察数据,在涉及长期维持特定治疗机制的治疗策略下进行反事实绩效评估。我们引入了一种使用人工删减和反概率加权的估算方法,其中包括创建一个验证数据集,模仿预测所依据的治疗策略。我们扩展了校准、区分度(c 指数和累积/动态 AUCt)和总体预测误差(布赖尔评分)的测量方法,以便评估反事实绩效。我们通过模拟研究对这些方法进行了评估,其中包括这些方法应能检测出性能不佳的情况。将我们的方法应用于肝脏移植表明,我们的程序可以量化支持器官分配关键决策的预测性能。
本文章由计算机程序翻译,如有差异,请以英文原文为准。
求助全文
约1分钟内获得全文 去求助
来源期刊
Epidemiology
Epidemiology 医学-公共卫生、环境卫生与职业卫生
CiteScore
6.70
自引率
3.70%
发文量
177
审稿时长
6-12 weeks
期刊介绍: Epidemiology publishes original research from all fields of epidemiology. The journal also welcomes review articles and meta-analyses, novel hypotheses, descriptions and applications of new methods, and discussions of research theory or public health policy. We give special consideration to papers from developing countries.
期刊最新文献
Adjusting adjustments: Using external data to estimate the impact of different confounder sets on published associations. Low-level PM2.5 exposure, Cardiovascular and Non-accidental Mortality, and Related Health Disparities in 12 U.S. States. A structural description of biases that generate immortal time. Associations Between Gestational Residential Radon Exposure and Term Low Birthweight in Connecticut, USA. Prenatal Exposure to Nonpersistent Chemicals and Fetal-to-childhood Growth Trajectories.
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
现在去查看 取消
×
提示
确定
0
微信
客服QQ
Book学术公众号 扫码关注我们
反馈
×
意见反馈
请填写您的意见或建议
请填写您的手机或邮箱
已复制链接
已复制链接
快去分享给好友吧!
我知道了
×
扫码分享
扫码分享
Book学术官方微信
Book学术文献互助
Book学术文献互助群
群 号:481959085
Book学术
文献互助 智能选刊 最新文献 互助须知 联系我们:info@booksci.cn
Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。
Copyright © 2023 Book学术 All rights reserved.
ghs 京公网安备 11010802042870号 京ICP备2023020795号-1