A Novel Metric for Measuring the Robustness of Large Language Models in Non-adversarial Scenarios

Samuel Ackerman, Ella Rabinovich, Eitan Farchi, Ateret Anaby-Tavor
{"title":"A Novel Metric for Measuring the Robustness of Large Language Models in Non-adversarial Scenarios","authors":"Samuel Ackerman, Ella Rabinovich, Eitan Farchi, Ateret Anaby-Tavor","doi":"arxiv-2408.01963","DOIUrl":null,"url":null,"abstract":"We evaluate the robustness of several large language models on multiple\ndatasets. Robustness here refers to the relative insensitivity of the model's\nanswers to meaning-preserving variants of their input. Benchmark datasets are\nconstructed by introducing naturally-occurring, non-malicious perturbations, or\nby generating semantically equivalent paraphrases of input questions or\nstatements. We further propose a novel metric for assessing a model robustness,\nand demonstrate its benefits in the non-adversarial scenario by empirical\nevaluation of several models on the created datasets.","PeriodicalId":501172,"journal":{"name":"arXiv - STAT - Applications","volume":null,"pages":null},"PeriodicalIF":0.0000,"publicationDate":"2024-08-04","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"arXiv - STAT - Applications","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/arxiv-2408.01963","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"","JCRName":"","Score":null,"Total":0}
引用次数: 0

Abstract

We evaluate the robustness of several large language models on multiple datasets. Robustness here refers to the relative insensitivity of the model's answers to meaning-preserving variants of their input. Benchmark datasets are constructed by introducing naturally-occurring, non-malicious perturbations, or by generating semantically equivalent paraphrases of input questions or statements. We further propose a novel metric for assessing a model robustness, and demonstrate its benefits in the non-adversarial scenario by empirical evaluation of several models on the created datasets.
查看原文
分享 分享
微信好友 朋友圈 QQ好友 复制链接
本刊更多论文
衡量非对抗场景下大型语言模型鲁棒性的新标准
我们评估了多个大型语言模型在多重数据集上的鲁棒性。这里的鲁棒性是指模型的答案对其输入的意义保留变体的相对不敏感性。基准数据集是通过引入自然发生的非恶意扰动,或生成输入问题或语句的语义等同解析来构建的。我们进一步提出了评估模型鲁棒性的新指标,并通过在创建的数据集上对多个模型进行实证评估,证明了该指标在非对抗性场景中的优势。
本文章由计算机程序翻译,如有差异,请以英文原文为准。
求助全文
约1分钟内获得全文 去求助
来源期刊
自引率
0.00%
发文量
0
期刊最新文献
A Bayesian framework to evaluate evidence in cases of alleged cheating with secret codes in sports Unsupervised anomaly detection in spatio-temporal stream network sensor data A Cost-Aware Approach to Adversarial Robustness in Neural Networks Teacher-student relationship and teaching styles in primary education. A model of analysis Monitoring road infrastructures from satellite images in Greater Maputo: an object-oriented classification approach
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
现在去查看 取消
×
提示
确定
0
微信
客服QQ
Book学术公众号 扫码关注我们
反馈
×
意见反馈
请填写您的意见或建议
请填写您的手机或邮箱
已复制链接
已复制链接
快去分享给好友吧!
我知道了
×
扫码分享
扫码分享
Book学术官方微信
Book学术文献互助
Book学术文献互助群
群 号:481959085
Book学术
文献互助 智能选刊 最新文献 互助须知 联系我们:info@booksci.cn
Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。
Copyright © 2023 Book学术 All rights reserved.
ghs 京公网安备 11010802042870号 京ICP备2023020795号-1