Cas Wognum, Jeremy R. Ash, Matteo Aldeghi, Raquel Rodríguez-Pérez, Cheng Fang, Alan C. Cheng, Daniel J. Price, Djork-Arné Clevert, Ola Engkvist, W. Patrick Walters
{"title":"A call for an industry-led initiative to critically assess machine learning for real-world drug discovery","authors":"Cas Wognum, Jeremy R. Ash, Matteo Aldeghi, Raquel Rodríguez-Pérez, Cheng Fang, Alan C. Cheng, Daniel J. Price, Djork-Arné Clevert, Ola Engkvist, W. Patrick Walters","doi":"10.1038/s42256-024-00911-w","DOIUrl":null,"url":null,"abstract":"<p>Machine learning (ML) is driving exciting innovations in drug discovery, but we need to be mindful of the circumstances that set this application apart. Unlike other fields with fit-for-purpose datasets consisting of millions of examples, published datasets in drug discovery are classically heterogeneous, imbalanced, noisy and expensive to generate<sup>1</sup>. Furthermore, the applications of ML in drug discovery are numerous, require familiarity with several scientific disciplines, and inform high-stakes decisions, such as expensive or time-consuming experiments. The absence of standardized, domain-appropriate datasets, guidelines and tools for the evaluation and comparison of methods has led to a growing gap between perceived progress and real-world impact, which is delaying the adoption of ML in drug discovery. To bridge this gap, we believe that the unique expertise of scientists in the industry, who operate in real-world contexts, will be essential in developing benchmarking protocols tailored to drug discovery. To that end, we already formed a unique collaboration between representatives from ten biotech and pharmaceutical companies, but we believe that an open-science, cross-industry and interdisciplinary effort is needed to tackle such grand challenges.</p><p>Fit-for-purpose benchmarks are powerful instruments to direct the ML community towards more impactful research and to unlock breakthrough results. The gold standard for unbiased evaluation is a blind, prospective benchmark, in which different methods are evaluated on a newly generated test set that will only be disclosed after the results have been announced. A popular example in drug discovery is CASP (Critical Assessment of Structure Prediction)<sup>2</sup>, which enabled a revolution in protein structure prediction by systematically identifying valuable innovations in the community<sup>3</sup>. However, data acquisition in drug discovery is expensive and time-consuming, limiting the accessibility and availability of blind benchmarks to the general research community.</p>","PeriodicalId":48533,"journal":{"name":"Nature Machine Intelligence","volume":null,"pages":null},"PeriodicalIF":18.8000,"publicationDate":"2024-10-04","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Nature Machine Intelligence","FirstCategoryId":"94","ListUrlMain":"https://doi.org/10.1038/s42256-024-00911-w","RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"COMPUTER SCIENCE, ARTIFICIAL INTELLIGENCE","Score":null,"Total":0}
引用次数: 0
Abstract
Machine learning (ML) is driving exciting innovations in drug discovery, but we need to be mindful of the circumstances that set this application apart. Unlike other fields with fit-for-purpose datasets consisting of millions of examples, published datasets in drug discovery are classically heterogeneous, imbalanced, noisy and expensive to generate1. Furthermore, the applications of ML in drug discovery are numerous, require familiarity with several scientific disciplines, and inform high-stakes decisions, such as expensive or time-consuming experiments. The absence of standardized, domain-appropriate datasets, guidelines and tools for the evaluation and comparison of methods has led to a growing gap between perceived progress and real-world impact, which is delaying the adoption of ML in drug discovery. To bridge this gap, we believe that the unique expertise of scientists in the industry, who operate in real-world contexts, will be essential in developing benchmarking protocols tailored to drug discovery. To that end, we already formed a unique collaboration between representatives from ten biotech and pharmaceutical companies, but we believe that an open-science, cross-industry and interdisciplinary effort is needed to tackle such grand challenges.
Fit-for-purpose benchmarks are powerful instruments to direct the ML community towards more impactful research and to unlock breakthrough results. The gold standard for unbiased evaluation is a blind, prospective benchmark, in which different methods are evaluated on a newly generated test set that will only be disclosed after the results have been announced. A popular example in drug discovery is CASP (Critical Assessment of Structure Prediction)2, which enabled a revolution in protein structure prediction by systematically identifying valuable innovations in the community3. However, data acquisition in drug discovery is expensive and time-consuming, limiting the accessibility and availability of blind benchmarks to the general research community.
期刊介绍:
Nature Machine Intelligence is a distinguished publication that presents original research and reviews on various topics in machine learning, robotics, and AI. Our focus extends beyond these fields, exploring their profound impact on other scientific disciplines, as well as societal and industrial aspects. We recognize limitless possibilities wherein machine intelligence can augment human capabilities and knowledge in domains like scientific exploration, healthcare, medical diagnostics, and the creation of safe and sustainable cities, transportation, and agriculture. Simultaneously, we acknowledge the emergence of ethical, social, and legal concerns due to the rapid pace of advancements.
To foster interdisciplinary discussions on these far-reaching implications, Nature Machine Intelligence serves as a platform for dialogue facilitated through Comments, News Features, News & Views articles, and Correspondence. Our goal is to encourage a comprehensive examination of these subjects.
Similar to all Nature-branded journals, Nature Machine Intelligence operates under the guidance of a team of skilled editors. We adhere to a fair and rigorous peer-review process, ensuring high standards of copy-editing and production, swift publication, and editorial independence.