{"title":"The difference of model robustness assessment using cross-validation and bootstrap methods","authors":"Rita Lasfar, Gergely Tóth","doi":"10.1002/cem.3530","DOIUrl":null,"url":null,"abstract":"<p>The validation principles on Quantitative Structure Activity Relationship issued by Organization for Economic and Co-operation and Development describe three criteria of model assessment: goodness of fit, robustness and prediction. In the case of robustness, two ways are possible as internal validation: bootstrap and cross-validation. We compared these validation metrics by checking their sample size dependence, rank correlations to other metrics and uncertainty. We used modeling methods from multivariate linear regression to artificial neural network on 14 open access datasets. We found that the metrics provide similar sample size dependence and correlation to other validation parameters. The individual uncertainty originating from the calculation recipes of the metrics is much smaller for both ways than the part caused by the selection of the training set or the training/test split. We concluded that the metrics of the two techniques are interchangeable, but the interpretation of cross-validation parameters is easier according to their similar range to goodness-of-fit and prediction metrics. Furthermore, the variance originating from the random elements of the calculation of cross-validation metrics is slightly smaller than those of bootstrap ones, if equal calculation load is applied.</p>","PeriodicalId":15274,"journal":{"name":"Journal of Chemometrics","volume":null,"pages":null},"PeriodicalIF":2.3000,"publicationDate":"2024-01-11","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Journal of Chemometrics","FirstCategoryId":"92","ListUrlMain":"https://onlinelibrary.wiley.com/doi/10.1002/cem.3530","RegionNum":4,"RegionCategory":"化学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"SOCIAL WORK","Score":null,"Total":0}
引用次数: 0
Abstract
The validation principles on Quantitative Structure Activity Relationship issued by Organization for Economic and Co-operation and Development describe three criteria of model assessment: goodness of fit, robustness and prediction. In the case of robustness, two ways are possible as internal validation: bootstrap and cross-validation. We compared these validation metrics by checking their sample size dependence, rank correlations to other metrics and uncertainty. We used modeling methods from multivariate linear regression to artificial neural network on 14 open access datasets. We found that the metrics provide similar sample size dependence and correlation to other validation parameters. The individual uncertainty originating from the calculation recipes of the metrics is much smaller for both ways than the part caused by the selection of the training set or the training/test split. We concluded that the metrics of the two techniques are interchangeable, but the interpretation of cross-validation parameters is easier according to their similar range to goodness-of-fit and prediction metrics. Furthermore, the variance originating from the random elements of the calculation of cross-validation metrics is slightly smaller than those of bootstrap ones, if equal calculation load is applied.
期刊介绍:
The Journal of Chemometrics is devoted to the rapid publication of original scientific papers, reviews and short communications on fundamental and applied aspects of chemometrics. It also provides a forum for the exchange of information on meetings and other news relevant to the growing community of scientists who are interested in chemometrics and its applications. Short, critical review papers are a particularly important feature of the journal, in view of the multidisciplinary readership at which it is aimed.