Majority of the current state-of-the-art test suite (TS) prioritization algorithms for black-box testing focus on improving average percentage of fault detected (APFD) metric. These however suffer from two critical challenges 1) high time complexity of $ge O(n^{2})$ where n is the number of test suites, and 2) limited ability to self-stop TS Prioritization (TSP) computation if the system under test (SUT) becomes highly stable. In this work we present an approach to overcome these two challenges while achieving high APFD efficiency over the conventional random ordering. A novel algorithm called Bayesian approach to regression testing (BART) is developed herein which models continuous integration (CI) cycle’s attributes like test suite life cycle (TSLC), stability and bugs as Bayesian inference pattern namely Dirichlet-Multinomial model. This work demonstrates that BART’s APFD metrics improve significantly in comparison to conventional random ordering and therefore this approach achieves for the first time a complexity of $O(nlogn)$ for black-box based test prioritization.
{"title":"Bayesian Approach for Regression Testing (BART) using Test Suite Prioritization","authors":"Prabuddh Gupta, Divya Balakrishna, Rohit R. Shende, Vikram Raina, Shalini Lal, Aditya Doshatti, Lalitha Sripada, Mitesh Sharma, Shiva Thamilavel","doi":"10.1109/STC55697.2022.00027","DOIUrl":"https://doi.org/10.1109/STC55697.2022.00027","url":null,"abstract":"Majority of the current state-of-the-art test suite (TS) prioritization algorithms for black-box testing focus on improving average percentage of fault detected (APFD) metric. These however suffer from two critical challenges 1) high time complexity of $ge O(n^{2})$ where n is the number of test suites, and 2) limited ability to self-stop TS Prioritization (TSP) computation if the system under test (SUT) becomes highly stable. In this work we present an approach to overcome these two challenges while achieving high APFD efficiency over the conventional random ordering. A novel algorithm called Bayesian approach to regression testing (BART) is developed herein which models continuous integration (CI) cycle’s attributes like test suite life cycle (TSLC), stability and bugs as Bayesian inference pattern namely Dirichlet-Multinomial model. This work demonstrates that BART’s APFD metrics improve significantly in comparison to conventional random ordering and therefore this approach achieves for the first time a complexity of $O(nlogn)$ for black-box based test prioritization.","PeriodicalId":170123,"journal":{"name":"2022 IEEE 29th Annual Software Technology Conference (STC)","volume":"74 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"116253643","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2022-10-01DOI: 10.1109/STC55697.2022.00031
R. Kuhn, Joshua D. Roberts, David F. Ferraiolo, J. Defranco
Industry continues to be challenged when attempting to share data among organizations, especially when the data comes from different database management systems (DBMS) and different DBMS schemas. Another concern is that privacy laws may require some types of data to be protected under local access policies. We describe a secure data sharing solution using Next-generation Database Access Control (NDAC) and the Data Block Matrix (DBM). A clinical trial data use case is discussed, as well as a description of a proof-of-concept implementation of the DBM using Hyperledger Fabric. The solution described allows data access where the data resides, rather than exchanging or being centrally stored. Additionally, it solves the conflict between conventional blockchain use and privacy regulations, by using a form of distributed ledger technology that meets ‘right to erasure’ requirements.
{"title":"A Distributed Ledger Technology Design using Hyperledger Fabric and a Clinical Trial Use Case","authors":"R. Kuhn, Joshua D. Roberts, David F. Ferraiolo, J. Defranco","doi":"10.1109/STC55697.2022.00031","DOIUrl":"https://doi.org/10.1109/STC55697.2022.00031","url":null,"abstract":"Industry continues to be challenged when attempting to share data among organizations, especially when the data comes from different database management systems (DBMS) and different DBMS schemas. Another concern is that privacy laws may require some types of data to be protected under local access policies. We describe a secure data sharing solution using Next-generation Database Access Control (NDAC) and the Data Block Matrix (DBM). A clinical trial data use case is discussed, as well as a description of a proof-of-concept implementation of the DBM using Hyperledger Fabric. The solution described allows data access where the data resides, rather than exchanging or being centrally stored. Additionally, it solves the conflict between conventional blockchain use and privacy regulations, by using a form of distributed ledger technology that meets ‘right to erasure’ requirements.","PeriodicalId":170123,"journal":{"name":"2022 IEEE 29th Annual Software Technology Conference (STC)","volume":"35 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"134271191","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2022-10-01DOI: 10.1109/STC55697.2022.00012
Sai Gurrapu, Lifu Huang, Feras A. Batarseh
With the advent of deep learning, text generation language models have improved dramatically, with text at a similar level as human-written text. This can lead to rampant misinformation because content can now be created cheaply and distributed quickly. Automated claim verification methods exist to validate claims, but they lack foundational data and often use mainstream news as evidence sources that are strongly biased towards a specific agenda. Current claim verification methods use deep neural network models and complex algorithms for a high classification accuracy but it is at the expense of model explainability. The models are black-boxes and their decision-making process and the steps it took to arrive at a final prediction are obfuscated from the user. We introduce a novel claim verification approach, namely: ExClaim, that attempts to provide an explainable claim verification system with foundational evidence. Inspired by the legal system, ExClaim leverages rationalization to provide a verdict for the claim and justifies the verdict through a natural language explanation (rationale) to describe the model’s decision-making process. ExClaim treats the verdict classification task as a question-answer problem and achieves a performance of 0.93 F1 score. It provides subtasks explanations to also justify the intermediate outcomes. Statistical and Explainable AI (XAI) evaluations are conducted to ensure valid and trustworthy outcomes. Ensuring claim verification systems are assured, rational, and explainable is an essential step toward improving Human-AI trust and the accessibility of black-box systems.
{"title":"ExClaim: Explainable Neural Claim Verification Using Rationalization","authors":"Sai Gurrapu, Lifu Huang, Feras A. Batarseh","doi":"10.1109/STC55697.2022.00012","DOIUrl":"https://doi.org/10.1109/STC55697.2022.00012","url":null,"abstract":"With the advent of deep learning, text generation language models have improved dramatically, with text at a similar level as human-written text. This can lead to rampant misinformation because content can now be created cheaply and distributed quickly. Automated claim verification methods exist to validate claims, but they lack foundational data and often use mainstream news as evidence sources that are strongly biased towards a specific agenda. Current claim verification methods use deep neural network models and complex algorithms for a high classification accuracy but it is at the expense of model explainability. The models are black-boxes and their decision-making process and the steps it took to arrive at a final prediction are obfuscated from the user. We introduce a novel claim verification approach, namely: ExClaim, that attempts to provide an explainable claim verification system with foundational evidence. Inspired by the legal system, ExClaim leverages rationalization to provide a verdict for the claim and justifies the verdict through a natural language explanation (rationale) to describe the model’s decision-making process. ExClaim treats the verdict classification task as a question-answer problem and achieves a performance of 0.93 F1 score. It provides subtasks explanations to also justify the intermediate outcomes. Statistical and Explainable AI (XAI) evaluations are conducted to ensure valid and trustworthy outcomes. Ensuring claim verification systems are assured, rational, and explainable is an essential step toward improving Human-AI trust and the accessibility of black-box systems.","PeriodicalId":170123,"journal":{"name":"2022 IEEE 29th Annual Software Technology Conference (STC)","volume":"50 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"133089647","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2022-10-01DOI: 10.1109/STC55697.2022.00032
P. Laplante, Rick Kuhn
Artificial intelligence (AI) systems are increasingly seen in many public facing applications such as self-driving land vehicles, autonomous aircraft, medical systems and financial systems. AI systems should equal or surpass human performance, but given the consequences of failure or erroneous or unfair decisions in these systems, how do we assure the public that these systems work as intended and will not cause harm? For example, that an autonomous vehicle does not crash or that intelligent credit scoring system is not biased, even after passing substantial acceptance testing prior to release. In this paper we discuss AI trust and assurance and related concepts, that is, assured autonomy, particularly for critical systems. Then we discuss how to establish trust through AI assurance activities throughout the system development lifecycle. Finally, we introduce a “trust but verify continuously” approach to AI assurance, which describes assured autonomy activities in a model based systems development context and includes postdelivery activities for continuous assurance.
{"title":"AI Assurance for the Public – Trust but Verify, Continuously","authors":"P. Laplante, Rick Kuhn","doi":"10.1109/STC55697.2022.00032","DOIUrl":"https://doi.org/10.1109/STC55697.2022.00032","url":null,"abstract":"Artificial intelligence (AI) systems are increasingly seen in many public facing applications such as self-driving land vehicles, autonomous aircraft, medical systems and financial systems. AI systems should equal or surpass human performance, but given the consequences of failure or erroneous or unfair decisions in these systems, how do we assure the public that these systems work as intended and will not cause harm? For example, that an autonomous vehicle does not crash or that intelligent credit scoring system is not biased, even after passing substantial acceptance testing prior to release. In this paper we discuss AI trust and assurance and related concepts, that is, assured autonomy, particularly for critical systems. Then we discuss how to establish trust through AI assurance activities throughout the system development lifecycle. Finally, we introduce a “trust but verify continuously” approach to AI assurance, which describes assured autonomy activities in a model based systems development context and includes postdelivery activities for continuous assurance.","PeriodicalId":170123,"journal":{"name":"2022 IEEE 29th Annual Software Technology Conference (STC)","volume":"60 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"115877711","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2022-10-01DOI: 10.1109/STC55697.2022.00014
P. Laplante, M. Kassab, J. Defranco
A challenging problem for software and systems engineers is to provide assurance of operations for a system that is critical but must operate in situations that cannot be easily created in the testing lab. For example, a space system cannot be fully tested in all operational modes until it is launched and nuclear power plants cannot be tested under real critical temperature overload conditions. This situation is particularly challenging when seeking to provide assurance in critical AI systems (CAIS) where the underlying algorithms may be very difficult to verify under any conditions. In these cases using systems that have a similar underlying application, operational profiles, user characteristics, and underlying AI algorithms may be suitable as testing proxies. For example, a robot vacuum may have significant operational and implementation similarities to act as a testing proxy for some aspects of an autonomous vehicle.In this work we discuss the challenges in assured autonomy for CAIS and suggest a way forward using proxy systems. We describe a methodology for characterizing CAIS and matching them to their non-critical proxy equivalent. Examples are given along with a discussion of the history of other kinds of proxy verification and validation
{"title":"Proxy Verification and Validation For Critical Autonomous and AI Systems","authors":"P. Laplante, M. Kassab, J. Defranco","doi":"10.1109/STC55697.2022.00014","DOIUrl":"https://doi.org/10.1109/STC55697.2022.00014","url":null,"abstract":"A challenging problem for software and systems engineers is to provide assurance of operations for a system that is critical but must operate in situations that cannot be easily created in the testing lab. For example, a space system cannot be fully tested in all operational modes until it is launched and nuclear power plants cannot be tested under real critical temperature overload conditions. This situation is particularly challenging when seeking to provide assurance in critical AI systems (CAIS) where the underlying algorithms may be very difficult to verify under any conditions. In these cases using systems that have a similar underlying application, operational profiles, user characteristics, and underlying AI algorithms may be suitable as testing proxies. For example, a robot vacuum may have significant operational and implementation similarities to act as a testing proxy for some aspects of an autonomous vehicle.In this work we discuss the challenges in assured autonomy for CAIS and suggest a way forward using proxy systems. We describe a methodology for characterizing CAIS and matching them to their non-critical proxy equivalent. Examples are given along with a discussion of the history of other kinds of proxy verification and validation","PeriodicalId":170123,"journal":{"name":"2022 IEEE 29th Annual Software Technology Conference (STC)","volume":"236 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"123365773","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2022-06-14DOI: 10.1109/STC55697.2022.00015
Tyler Cody
This paper considers key challenges to using re-inforcement learning (RL) with attack graphs to automate penetration testing in real-world applications from a systems perspective. RL approaches to automated penetration testing are actively being developed, but there is no consensus view on the representation of computer networks with which RL should be interacting. Moreover, there are significant open challenges to how those representations can be grounded to the real networks where RL solution methods are applied. This paper elaborates on representation and grounding using topic challenges of interacting with real networks in real-time, emulating realistic adversary behavior, and handling unstable, evolving networks. These challenges are both practical and mathematical, and they directly concern the reliability and dependability of penetration testing systems. This paper proposes a layered reference model to help organize related research and engineering efforts. The presented layered reference model contrasts traditional models of attack graph workflows because it is not scoped to a sequential, feed-forward generation and analysis process, but to broader aspects of lifecycle and continuous deployment. Researchers and practitioners can use the presented layered reference model as a first-principles outline to help orient the systems engineering of their penetration testing systems.
{"title":"A Layered Reference Model for Penetration Testing with Reinforcement Learning and Attack Graphs","authors":"Tyler Cody","doi":"10.1109/STC55697.2022.00015","DOIUrl":"https://doi.org/10.1109/STC55697.2022.00015","url":null,"abstract":"This paper considers key challenges to using re-inforcement learning (RL) with attack graphs to automate penetration testing in real-world applications from a systems perspective. RL approaches to automated penetration testing are actively being developed, but there is no consensus view on the representation of computer networks with which RL should be interacting. Moreover, there are significant open challenges to how those representations can be grounded to the real networks where RL solution methods are applied. This paper elaborates on representation and grounding using topic challenges of interacting with real networks in real-time, emulating realistic adversary behavior, and handling unstable, evolving networks. These challenges are both practical and mathematical, and they directly concern the reliability and dependability of penetration testing systems. This paper proposes a layered reference model to help organize related research and engineering efforts. The presented layered reference model contrasts traditional models of attack graph workflows because it is not scoped to a sequential, feed-forward generation and analysis process, but to broader aspects of lifecycle and continuous deployment. Researchers and practitioners can use the presented layered reference model as a first-principles outline to help orient the systems engineering of their penetration testing systems.","PeriodicalId":170123,"journal":{"name":"2022 IEEE 29th Annual Software Technology Conference (STC)","volume":"45 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-06-14","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"115644886","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2021-09-23DOI: 10.1109/STC55697.2022.00033
R. NarayanaaS, M. Sivaranjan, S. LekshmiR
Fault localization is an imperative method in fault tolerance in a distributed environment that designs a blueprint for continuing the ongoing process even when one or many modules are non-functional. Visualizing a distributed environment as a graph, whose nodes represent faults (fault graph), allows us to introduce probabilistic weights to both edges and nodes that cause the faults. With multiple modules like databases, run-time cloud, etc. making up a distributed environment and extensively, a cloud environment, we aim to address the problem of optimally and accurately performing fault localization in a distributed environment by modifying the Graph optimization approach to localization and centrality, specific to fault graphs.
{"title":"Fault Localization in Cloud using Centrality Measures","authors":"R. NarayanaaS, M. Sivaranjan, S. LekshmiR","doi":"10.1109/STC55697.2022.00033","DOIUrl":"https://doi.org/10.1109/STC55697.2022.00033","url":null,"abstract":"Fault localization is an imperative method in fault tolerance in a distributed environment that designs a blueprint for continuing the ongoing process even when one or many modules are non-functional. Visualizing a distributed environment as a graph, whose nodes represent faults (fault graph), allows us to introduce probabilistic weights to both edges and nodes that cause the faults. With multiple modules like databases, run-time cloud, etc. making up a distributed environment and extensively, a cloud environment, we aim to address the problem of optimally and accurately performing fault localization in a distributed environment by modifying the Graph optimization approach to localization and centrality, specific to fault graphs.","PeriodicalId":170123,"journal":{"name":"2022 IEEE 29th Annual Software Technology Conference (STC)","volume":"25 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-09-23","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"125016158","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
京公网安备 11010802042870号
京ICP备2023020795号-1
扫码关注我们
求助内容：
应助结果提醒方式：