A Multi-layered Collaborative Framework for Evidence-driven Data Requirements Engineering for Machine Learning-based Safety-critical Systems

IF 0.4 Q4 COMPUTER SCIENCE, INFORMATION SYSTEMS Applied Computing Review Pub Date : 2023-03-27 DOI:10.1145/3555776.3577647
Sangeeta Dey, Seok-Won Lee
{"title":"A Multi-layered Collaborative Framework for Evidence-driven Data Requirements Engineering for Machine Learning-based Safety-critical Systems","authors":"Sangeeta Dey, Seok-Won Lee","doi":"10.1145/3555776.3577647","DOIUrl":null,"url":null,"abstract":"In the days of AI, data-centric machine learning (ML) models are increasingly used in various complex systems. While many researchers are focusing on specifying ML-specific performance requirements, not enough guideline is provided to engineer the data requirements systematically involving diverse stakeholders. Lack of written agreement about the training data, collaboration bottlenecks, lack of data validation framework, etc. are posing new challenges to ensuring training data fitness for safety-critical ML components. To reduce these gaps, we propose a multi-layered framework that helps to perceive and elicit data requirements. We provide a template for verifiable data requirements specifications. Moreover, we show how such requirements can facilitate an evidence-driven assessment of the training data quality based on the experts' judgments about the satisfaction of the requirements. We use Dempster Shafer's theory to combine experts' subjective opinions in the process. A preliminary case study on the CityPersons dataset for the pedestrian detection feature of autonomous cars shows the usefulness of the proposed framework for data requirements understanding and the confidence assessment of the dataset.","PeriodicalId":42971,"journal":{"name":"Applied Computing Review","volume":null,"pages":null},"PeriodicalIF":0.4000,"publicationDate":"2023-03-27","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Applied Computing Review","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/10.1145/3555776.3577647","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q4","JCRName":"COMPUTER SCIENCE, INFORMATION SYSTEMS","Score":null,"Total":0}
引用次数: 0

Abstract

In the days of AI, data-centric machine learning (ML) models are increasingly used in various complex systems. While many researchers are focusing on specifying ML-specific performance requirements, not enough guideline is provided to engineer the data requirements systematically involving diverse stakeholders. Lack of written agreement about the training data, collaboration bottlenecks, lack of data validation framework, etc. are posing new challenges to ensuring training data fitness for safety-critical ML components. To reduce these gaps, we propose a multi-layered framework that helps to perceive and elicit data requirements. We provide a template for verifiable data requirements specifications. Moreover, we show how such requirements can facilitate an evidence-driven assessment of the training data quality based on the experts' judgments about the satisfaction of the requirements. We use Dempster Shafer's theory to combine experts' subjective opinions in the process. A preliminary case study on the CityPersons dataset for the pedestrian detection feature of autonomous cars shows the usefulness of the proposed framework for data requirements understanding and the confidence assessment of the dataset.
查看原文
分享 分享
微信好友 朋友圈 QQ好友 复制链接
本刊更多论文
基于机器学习的安全关键系统证据驱动数据需求工程的多层协作框架
在人工智能时代,以数据为中心的机器学习(ML)模型越来越多地用于各种复杂系统。虽然许多研究人员专注于指定特定于ml的性能需求,但没有提供足够的指南来系统地设计涉及不同利益相关者的数据需求。缺乏关于训练数据的书面协议、协作瓶颈、缺乏数据验证框架等,都对确保训练数据适合安全关键的ML组件构成了新的挑战。为了减少这些差距,我们提出了一个多层框架来帮助感知和引出数据需求。我们为可验证的数据需求规范提供了一个模板。此外,我们展示了这些需求如何能够促进基于专家对需求满意度的判断的训练数据质量的证据驱动评估。我们运用Dempster Shafer的理论,结合专家的主观意见。对自动驾驶汽车行人检测特征的CityPersons数据集的初步案例研究表明,所提出的框架对于数据需求理解和数据集置信度评估的有用性。
本文章由计算机程序翻译,如有差异,请以英文原文为准。
求助全文
约1分钟内获得全文 去求助
来源期刊
Applied Computing Review
Applied Computing Review COMPUTER SCIENCE, INFORMATION SYSTEMS-
自引率
40.00%
发文量
8
期刊最新文献
DIWS-LCR-Rot-hop++: A Domain-Independent Word Selector for Cross-Domain Aspect-Based Sentiment Classification Leveraging Semantic Technologies for Collaborative Inference of Threatening IoT Dependencies Relating Optimal Repairs in Ontology Engineering with Contraction Operations in Belief Change Block-RACS: Towards Reputation-Aware Client Selection and Monetization Mechanism for Federated Learning Elastic Data Binning: Time-Series Sketching for Time-Domain Astrophysics Analysis
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
现在去查看 取消
×
提示
确定
0
微信
客服QQ
Book学术公众号 扫码关注我们
反馈
×
意见反馈
请填写您的意见或建议
请填写您的手机或邮箱
已复制链接
已复制链接
快去分享给好友吧!
我知道了
×
扫码分享
扫码分享
Book学术官方微信
Book学术文献互助
Book学术文献互助群
群 号:481959085
Book学术
文献互助 智能选刊 最新文献 互助须知 联系我们:info@booksci.cn
Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。
Copyright © 2023 Book学术 All rights reserved.
ghs 京公网安备 11010802042870号 京ICP备2023020795号-1