The difference of model robustness assessment using cross-validation and bootstrap methods

IF 2.3 4区 化学 Q1 SOCIAL WORK Journal of Chemometrics Pub Date : 2024-01-11 DOI:10.1002/cem.3530
Rita Lasfar, Gergely Tóth
{"title":"The difference of model robustness assessment using cross-validation and bootstrap methods","authors":"Rita Lasfar,&nbsp;Gergely Tóth","doi":"10.1002/cem.3530","DOIUrl":null,"url":null,"abstract":"<p>The validation principles on Quantitative Structure Activity Relationship issued by Organization for Economic and Co-operation and Development describe three criteria of model assessment: goodness of fit, robustness and prediction. In the case of robustness, two ways are possible as internal validation: bootstrap and cross-validation. We compared these validation metrics by checking their sample size dependence, rank correlations to other metrics and uncertainty. We used modeling methods from multivariate linear regression to artificial neural network on 14 open access datasets. We found that the metrics provide similar sample size dependence and correlation to other validation parameters. The individual uncertainty originating from the calculation recipes of the metrics is much smaller for both ways than the part caused by the selection of the training set or the training/test split. We concluded that the metrics of the two techniques are interchangeable, but the interpretation of cross-validation parameters is easier according to their similar range to goodness-of-fit and prediction metrics. Furthermore, the variance originating from the random elements of the calculation of cross-validation metrics is slightly smaller than those of bootstrap ones, if equal calculation load is applied.</p>","PeriodicalId":15274,"journal":{"name":"Journal of Chemometrics","volume":null,"pages":null},"PeriodicalIF":2.3000,"publicationDate":"2024-01-11","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Journal of Chemometrics","FirstCategoryId":"92","ListUrlMain":"https://onlinelibrary.wiley.com/doi/10.1002/cem.3530","RegionNum":4,"RegionCategory":"化学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"SOCIAL WORK","Score":null,"Total":0}
引用次数: 0

Abstract

The validation principles on Quantitative Structure Activity Relationship issued by Organization for Economic and Co-operation and Development describe three criteria of model assessment: goodness of fit, robustness and prediction. In the case of robustness, two ways are possible as internal validation: bootstrap and cross-validation. We compared these validation metrics by checking their sample size dependence, rank correlations to other metrics and uncertainty. We used modeling methods from multivariate linear regression to artificial neural network on 14 open access datasets. We found that the metrics provide similar sample size dependence and correlation to other validation parameters. The individual uncertainty originating from the calculation recipes of the metrics is much smaller for both ways than the part caused by the selection of the training set or the training/test split. We concluded that the metrics of the two techniques are interchangeable, but the interpretation of cross-validation parameters is easier according to their similar range to goodness-of-fit and prediction metrics. Furthermore, the variance originating from the random elements of the calculation of cross-validation metrics is slightly smaller than those of bootstrap ones, if equal calculation load is applied.

Abstract Image

Abstract Image

查看原文
分享 分享
微信好友 朋友圈 QQ好友 复制链接
本刊更多论文
使用交叉验证法和引导法评估模型稳健性的区别
经济合作与发展组织发布的《定量结构活动关系验证原则》描述了模型评估的三个标准:拟合度、稳健性和预测。在稳健性方面,有两种内部验证方法:自举法和交叉验证。我们通过检查这些验证指标的样本量依赖性、与其他指标的等级相关性和不确定性,对它们进行了比较。我们在 14 个开放数据集上使用了从多元线性回归到人工神经网络的建模方法。我们发现,这些指标提供了类似的样本大小依赖性以及与其他验证参数的相关性。在这两种方法中,源于度量标准计算配方的个别不确定性要比源于训练集选择或训练/测试分割的部分小得多。我们的结论是,这两种技术的度量标准可以互换,但交叉验证参数的解释更容易,因为它们与拟合优度和预测度量标准的范围相似。此外,在计算负荷相同的情况下,交叉验证指标计算中随机因素产生的方差略小于 bootstrap 指标。
本文章由计算机程序翻译,如有差异,请以英文原文为准。
求助全文
约1分钟内获得全文 去求助
来源期刊
Journal of Chemometrics
Journal of Chemometrics 化学-分析化学
CiteScore
5.20
自引率
8.30%
发文量
78
审稿时长
2 months
期刊介绍: The Journal of Chemometrics is devoted to the rapid publication of original scientific papers, reviews and short communications on fundamental and applied aspects of chemometrics. It also provides a forum for the exchange of information on meetings and other news relevant to the growing community of scientists who are interested in chemometrics and its applications. Short, critical review papers are a particularly important feature of the journal, in view of the multidisciplinary readership at which it is aimed.
期刊最新文献
Issue Information Issue Information Resampling as a Robust Measure of Model Complexity in PARAFAC Models Population Power Curves in ASCA With Permutation Testing A Non‐Linear Model for Multiple Alcohol Intakes and Optimal Designs Strategies
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
现在去查看 取消
×
提示
确定
0
微信
客服QQ
Book学术公众号 扫码关注我们
反馈
×
意见反馈
请填写您的意见或建议
请填写您的手机或邮箱
已复制链接
已复制链接
快去分享给好友吧!
我知道了
×
扫码分享
扫码分享
Book学术官方微信
Book学术文献互助
Book学术文献互助群
群 号:481959085
Book学术
文献互助 智能选刊 最新文献 互助须知 联系我们:info@booksci.cn
Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。
Copyright © 2023 Book学术 All rights reserved.
ghs 京公网安备 11010802042870号 京ICP备2023020795号-1