Measures and Metrics of ML Data and Models to Assure Reliable and Safe Systems

Benjamin Werner, Benjamin Schumeg, Jon Vigil, Shane N. Hall, Benjamin G. Thengvall, Mikel D. Petty
{"title":"Measures and Metrics of ML Data and Models to Assure Reliable and Safe Systems","authors":"Benjamin Werner, Benjamin Schumeg, Jon Vigil, Shane N. Hall, Benjamin G. Thengvall, Mikel D. Petty","doi":"10.1109/RAMS51492.2024.10457615","DOIUrl":null,"url":null,"abstract":"The US Army solicited partners through a Broad Agency Announcement to propose solutions under a Small Business Technology Transfer contract mechanism for the program “Metrics and Methods for Verification, Validation, Assurance and Trust of Machine Learning Models & Data for Safety-Critical Applications in Armaments Systems.” OptTek Systems, Inc. and University of Alabama in Huntsville (UAH) were one of the selected proposals for Phase I. Under this contract agreement OptTek and UAH set the goal to research & develop (R&D) fundamental metrics & measures for the certification & qualification of ML training data sets & models. Of particular note, the use of a safety score calculated from the accuracy as well as a dedicated look at data quality have been demonstrated as reasonable approaches to the proposed topic. As the Technical Point of Contact for this effort, the US Army Combat Capabilities Development Command Armaments Center (DEVCOM AC) authored the topic and provided guidance on the effort to align with mission objectives. This paper is an exploration of the research and development conducted by OptTek and UAH within the framework of how it may be applied to the assurance of systems to be developed by the US Army and augment practices in reliability and safety.","PeriodicalId":518362,"journal":{"name":"2024 Annual Reliability and Maintainability Symposium (RAMS)","volume":"15 9","pages":"1-6"},"PeriodicalIF":0.0000,"publicationDate":"2024-01-22","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"2024 Annual Reliability and Maintainability Symposium (RAMS)","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/10.1109/RAMS51492.2024.10457615","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"","JCRName":"","Score":null,"Total":0}
引用次数: 0

Abstract

The US Army solicited partners through a Broad Agency Announcement to propose solutions under a Small Business Technology Transfer contract mechanism for the program “Metrics and Methods for Verification, Validation, Assurance and Trust of Machine Learning Models & Data for Safety-Critical Applications in Armaments Systems.” OptTek Systems, Inc. and University of Alabama in Huntsville (UAH) were one of the selected proposals for Phase I. Under this contract agreement OptTek and UAH set the goal to research & develop (R&D) fundamental metrics & measures for the certification & qualification of ML training data sets & models. Of particular note, the use of a safety score calculated from the accuracy as well as a dedicated look at data quality have been demonstrated as reasonable approaches to the proposed topic. As the Technical Point of Contact for this effort, the US Army Combat Capabilities Development Command Armaments Center (DEVCOM AC) authored the topic and provided guidance on the effort to align with mission objectives. This paper is an exploration of the research and development conducted by OptTek and UAH within the framework of how it may be applied to the assurance of systems to be developed by the US Army and augment practices in reliability and safety.
查看原文
分享 分享
微信好友 朋友圈 QQ好友 复制链接
本刊更多论文
确保系统可靠安全的 ML 数据和模型的衡量标准和指标
美国陆军通过 "广泛机构公告"(Broad Agency Announcement)征集合作伙伴,在小企业技术转让合同机制下为 "用于军备系统安全关键应用的机器学习模型和数据的验证、确认、保证和信任的度量标准和方法 "项目提出解决方案。根据该合同协议,OptTek Systems 公司和阿拉巴马大学亨茨维尔分校(UAH)的目标是研究和开发(R&D)用于 ML 训练数据集和模型认证和鉴定的基本指标和措施。特别值得注意的是,根据准确性计算的安全分数以及对数据质量的专门研究已被证明是解决拟议主题的合理方法。作为这项工作的技术联络点,美国陆军作战能力发展司令部军备中心(DEVCOM AC)撰写了这一课题,并提供了与任务目标相一致的工作指导。本文探讨了 OptTek 和 UAH 在如何将其应用于美国陆军即将开发的系统保证以及增强可靠性和安全性实践的框架内进行的研究和开发。
本文章由计算机程序翻译,如有差异,请以英文原文为准。
求助全文
约1分钟内获得全文 去求助
来源期刊
自引率
0.00%
发文量
0
期刊最新文献
Do Bayesian Neural Networks Weapon System Improve Predictive Maintenance? Measures and Metrics of ML Data and Models to Assure Reliable and Safe Systems Estimating Multi-Phase Life Cycle Reliability by Combining Failure Rate Adjustments into Duty Cycle An Automated Performance Evaluation of the Newborn Life Support Procedure Maximum Likelihood Estimation with the Two-Parameter Exponential Model and Interval-Censored Data
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
现在去查看 取消
×
提示
确定
0
微信
客服QQ
Book学术公众号 扫码关注我们
反馈
×
意见反馈
请填写您的意见或建议
请填写您的手机或邮箱
已复制链接
已复制链接
快去分享给好友吧!
我知道了
×
扫码分享
扫码分享
Book学术官方微信
Book学术文献互助
Book学术文献互助群
群 号:481959085
Book学术
文献互助 智能选刊 最新文献 互助须知 联系我们:info@booksci.cn
Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。
Copyright © 2023 Book学术 All rights reserved.
ghs 京公网安备 11010802042870号 京ICP备2023020795号-1