Pub Date : 2022-10-01DOI: 10.1109/ISSREW55968.2022.00070
Qingyang Zhang, F. Machida, E. Andrade
Computing in drones has recently become popular for various real-world applications. To assure the performance and reliability of drone computing, systems can also adopt computation offloading to a nearby fog or edge server through a wireless network. As the offloading performance is significantly affected by the amount of workload, the network stability, and the competing use of a shared resource, performance estimation is essential for such systems. In this paper, we analyze the performance bottleneck of a drone system consisting of multiple drones that offload the tasks to a shared fog node. We investigate how resource conflict due to computation offloading causes the performance bottleneck of the drone computation system. To model the behavior of the system and analyze the performance and availability, we use Stochastic Reward Nets (SRN s). Through the numerical experiments, we confirm that the benefit of computation offloading deteriorates as the number of competing drones increases. To overcome the performance bottleneck, we also discuss potential solutions to mitigate the issue of a shared fog node.
{"title":"Performance Bottleneck Analysis of Drone Computation Offloading to a Shared Fog Node","authors":"Qingyang Zhang, F. Machida, E. Andrade","doi":"10.1109/ISSREW55968.2022.00070","DOIUrl":"https://doi.org/10.1109/ISSREW55968.2022.00070","url":null,"abstract":"Computing in drones has recently become popular for various real-world applications. To assure the performance and reliability of drone computing, systems can also adopt computation offloading to a nearby fog or edge server through a wireless network. As the offloading performance is significantly affected by the amount of workload, the network stability, and the competing use of a shared resource, performance estimation is essential for such systems. In this paper, we analyze the performance bottleneck of a drone system consisting of multiple drones that offload the tasks to a shared fog node. We investigate how resource conflict due to computation offloading causes the performance bottleneck of the drone computation system. To model the behavior of the system and analyze the performance and availability, we use Stochastic Reward Nets (SRN s). Through the numerical experiments, we confirm that the benefit of computation offloading deteriorates as the number of competing drones increases. To overcome the performance bottleneck, we also discuss potential solutions to mitigate the issue of a shared fog node.","PeriodicalId":178302,"journal":{"name":"2022 IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW)","volume":"78 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"114377706","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2022-10-01DOI: 10.1109/ISSREW55968.2022.00089
D. Drusinsky, J. Michael, Matthew Litton
Machine learning classifiers can be used as speci-fications for runtime monitoring (RM), which in turn supports evaluating autonomous systems during design-time and detecting/responding to exceptional situations during system operation. In this paper we describe how the use of machine-learned specifications enhances the effectiveness of RM for verification and validation (V & V) of autonomous cyberphysical systems (CPSs). In addition, we show that the development of machine-learned specifications has a predictable cost, at less than $100 per specification, using 2022 cloud computing pricing. Finally, a key benefit of our approach is that developing specifications by training ML models brings the task of developing robust specifications from the realm of doctoral-level experts into the domain of system developers and engineers.
{"title":"Machine-Learned Specifications for the Verification and Validation of Autonomous Cyberphysical Systems","authors":"D. Drusinsky, J. Michael, Matthew Litton","doi":"10.1109/ISSREW55968.2022.00089","DOIUrl":"https://doi.org/10.1109/ISSREW55968.2022.00089","url":null,"abstract":"Machine learning classifiers can be used as speci-fications for runtime monitoring (RM), which in turn supports evaluating autonomous systems during design-time and detecting/responding to exceptional situations during system operation. In this paper we describe how the use of machine-learned specifications enhances the effectiveness of RM for verification and validation (V & V) of autonomous cyberphysical systems (CPSs). In addition, we show that the development of machine-learned specifications has a predictable cost, at less than $100 per specification, using 2022 cloud computing pricing. Finally, a key benefit of our approach is that developing specifications by training ML models brings the task of developing robust specifications from the realm of doctoral-level experts into the domain of system developers and engineers.","PeriodicalId":178302,"journal":{"name":"2022 IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW)","volume":"48 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"121904095","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2022-10-01DOI: 10.1109/ISSREW55968.2022.00046
James J. Cusick, Alberto Avritzer, Allen Tse, Andrea Janes
We present an industrial experience report of the application of automated dependability assessment to a major Digital Transformation Initiative. This effort involved significant investment in the development, automation and visualization of the Non-Functional Requirements (NFRs). We present the details around the objectives of the NFR effort, challenges, the technical approach adopted, summary of results, and lessons learned. A clear description of steps which worked well are provided as well as the challenges which were found in meeting a wide range of technical and organizational goals in this process. A focus on the methods and results of developing NFRs within a DevOps environment and across a large heterogeneous computing platform are emphasized.
{"title":"Automated Dependability Assessment in DevOps Environments","authors":"James J. Cusick, Alberto Avritzer, Allen Tse, Andrea Janes","doi":"10.1109/ISSREW55968.2022.00046","DOIUrl":"https://doi.org/10.1109/ISSREW55968.2022.00046","url":null,"abstract":"We present an industrial experience report of the application of automated dependability assessment to a major Digital Transformation Initiative. This effort involved significant investment in the development, automation and visualization of the Non-Functional Requirements (NFRs). We present the details around the objectives of the NFR effort, challenges, the technical approach adopted, summary of results, and lessons learned. A clear description of steps which worked well are provided as well as the challenges which were found in meeting a wide range of technical and organizational goals in this process. A focus on the methods and results of developing NFRs within a DevOps environment and across a large heterogeneous computing platform are emphasized.","PeriodicalId":178302,"journal":{"name":"2022 IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW)","volume":"56 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"126982001","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2022-10-01DOI: 10.1109/ISSREW55968.2022.00093
Sayan Mukherjee, J. Rupe, J. Zhu
Explainable AI (XAI) is a topic of intense activity in the research community today. However, for AI models deployed in the critical infrastructure of communications networks, explainability alone is not enough to earn the trust of network operations teams comprising human experts with many decades of collective experience. In the present work we discuss some use cases in communications networks and state some of the additional properties, including accountability, that XAI models would have to satisfy before they can be widely deployed. In particular, we advocate for a human-in-the-Ioop approach to train and validate XAI models. Additionally, we discuss the use cases of XAI models around improving data preprocessing and data augmentation techniques, and refining data labeling rules for producing consistently labeled network datasets.
可解释人工智能(XAI)是当今研究界的一个热门话题。然而，对于部署在通信网络关键基础设施中的人工智能模型，仅凭可解释性不足以赢得由具有数十年集体经验的人类专家组成的网络运营团队的信任。在目前的工作中，我们讨论了通信网络中的一些用例，并说明了XAI模型在被广泛部署之前必须满足的一些附加属性，包括责任。特别地，我们提倡使用human- In -the- loop方法来训练和验证XAI模型。此外，我们还讨论了XAI模型的用例，围绕改进数据预处理和数据增强技术，以及改进数据标记规则以生成一致标记的网络数据集。
{"title":"XAI for Communication Networks","authors":"Sayan Mukherjee, J. Rupe, J. Zhu","doi":"10.1109/ISSREW55968.2022.00093","DOIUrl":"https://doi.org/10.1109/ISSREW55968.2022.00093","url":null,"abstract":"Explainable AI (XAI) is a topic of intense activity in the research community today. However, for AI models deployed in the critical infrastructure of communications networks, explainability alone is not enough to earn the trust of network operations teams comprising human experts with many decades of collective experience. In the present work we discuss some use cases in communications networks and state some of the additional properties, including accountability, that XAI models would have to satisfy before they can be widely deployed. In particular, we advocate for a human-in-the-Ioop approach to train and validate XAI models. Additionally, we discuss the use cases of XAI models around improving data preprocessing and data augmentation techniques, and refining data labeling rules for producing consistently labeled network datasets.","PeriodicalId":178302,"journal":{"name":"2022 IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW)","volume":"1 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"124512584","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2022-10-01DOI: 10.1109/ISSREW55968.2022.00055
Yiqun Chen, M. Bradbury, N. Suri
Fuzzing is an automated testing technique that utilizes injection of random inputs in a target program to help uncover vulnerabilities. Performance fuzzing extends the classic fuzzing approach and generates inputs that trigger poor performance. During our evaluation of performance fuzzing tools, we have identified certain conventionally used assumptions that do not always hold true. Our research (re)evaluates PERFFUZZ [1] in order to identify the limitations of current techniques, and guide the direction of future work for improvements to performance fuzzing. Our experimental results highlight two specific limitations. Firstly, we identify the assumption that the length of execution paths correlate to program performance is not always the case, and thus cannot reflect the quality of test cases generated by performance fuzzing. Secondly, the default testing parameters by the fuzzing process (timeouts and size limits) overly confine the input search space. Based on these observations, we suggest further investigation on performance fuzzing guidance, as well as controlled fuzzing and testing parameters.
{"title":"Towards Effective Performance Fuzzing","authors":"Yiqun Chen, M. Bradbury, N. Suri","doi":"10.1109/ISSREW55968.2022.00055","DOIUrl":"https://doi.org/10.1109/ISSREW55968.2022.00055","url":null,"abstract":"Fuzzing is an automated testing technique that utilizes injection of random inputs in a target program to help uncover vulnerabilities. Performance fuzzing extends the classic fuzzing approach and generates inputs that trigger poor performance. During our evaluation of performance fuzzing tools, we have identified certain conventionally used assumptions that do not always hold true. Our research (re)evaluates PERFFUZZ [1] in order to identify the limitations of current techniques, and guide the direction of future work for improvements to performance fuzzing. Our experimental results highlight two specific limitations. Firstly, we identify the assumption that the length of execution paths correlate to program performance is not always the case, and thus cannot reflect the quality of test cases generated by performance fuzzing. Secondly, the default testing parameters by the fuzzing process (timeouts and size limits) overly confine the input search space. Based on these observations, we suggest further investigation on performance fuzzing guidance, as well as controlled fuzzing and testing parameters.","PeriodicalId":178302,"journal":{"name":"2022 IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW)","volume":"26 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"127719026","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2022-10-01DOI: 10.1109/ISSREW55968.2022.00069
Philipp Schleiss, Yuki Hagiwara, Iwo Kurzidem, Francesco Carella
Deep learning (DL) is seen as an inevitable building block for perceiving the environment with sufficient detail and accuracy as required by automated driving functions. Despite this, its black-box nature and the therewith intertwined unpredictability still hinders its use in safety-critical systems. As such, this work addresses the problem of making this seemingly unpredictable nature measurable by providing a risk-based verification strategy, such as required by ISO 21448. In detail, a method is developed to break down acceptable risk into quantitative performance targets of individual DL-based components along the perception architecture. To verify these targets, the DL input space is split into areas according to the dimensions of a fine-grained operational design domain $(mu mathbf{ODD})$. As it is not feasible to reach full test coverage, the strategy suggests to distribute test efforts across these areas according to the associated risk. Moreover, the testing approach provides answers with respect to how much test coverage and confidence in the test result is required and how these figures relate to safety integrity levels (SILs).
{"title":"Towards the Quantitative Verification of Deep Learning for Safe Perception","authors":"Philipp Schleiss, Yuki Hagiwara, Iwo Kurzidem, Francesco Carella","doi":"10.1109/ISSREW55968.2022.00069","DOIUrl":"https://doi.org/10.1109/ISSREW55968.2022.00069","url":null,"abstract":"Deep learning (DL) is seen as an inevitable building block for perceiving the environment with sufficient detail and accuracy as required by automated driving functions. Despite this, its black-box nature and the therewith intertwined unpredictability still hinders its use in safety-critical systems. As such, this work addresses the problem of making this seemingly unpredictable nature measurable by providing a risk-based verification strategy, such as required by ISO 21448. In detail, a method is developed to break down acceptable risk into quantitative performance targets of individual DL-based components along the perception architecture. To verify these targets, the DL input space is split into areas according to the dimensions of a fine-grained operational design domain $(mu mathbf{ODD})$. As it is not feasible to reach full test coverage, the strategy suggests to distribute test efforts across these areas according to the associated risk. Moreover, the testing approach provides answers with respect to how much test coverage and confidence in the test result is required and how these figures relate to safety integrity levels (SILs).","PeriodicalId":178302,"journal":{"name":"2022 IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW)","volume":"9 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"125968972","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2022-10-01DOI: 10.1109/ISSREW55968.2022.00045
Oussama Jebbar, F. Khendek, M. Toeroe
Live testing is about testing a subsystem in production without causing any unacceptable disturbance to the production traffic. A subsystem is tested in production for multiple purposes such as deployment verification, fault prediction, fault localization, etc. The main challenge of live testing is alleviating the risk of test interferences as it may lead to a violation of a system's functional or non-functional requirements. To properly handle this risk, one needs to know which components present a risk of test interferences and what is the cost of the countermeasures to handle that risk. Existing literature relies heavily on human judgement, which can be time consuming, not always feasible, may provide misleading insight. In this paper we go through the challenges of automating this evaluation process and propose a solution to overcome them. Our solution consists of a method for components evaluation which goes through three steps, evaluation of test interferences that may manifest in external behaviour, evaluation of test interferences that may manifest in resource consumption, and finally the evaluation of the cost of implementing the countermeasures to overcome the risk of test interferences.
{"title":"A Method for Component Evaluation for Live Testing of Cloud Systems","authors":"Oussama Jebbar, F. Khendek, M. Toeroe","doi":"10.1109/ISSREW55968.2022.00045","DOIUrl":"https://doi.org/10.1109/ISSREW55968.2022.00045","url":null,"abstract":"Live testing is about testing a subsystem in production without causing any unacceptable disturbance to the production traffic. A subsystem is tested in production for multiple purposes such as deployment verification, fault prediction, fault localization, etc. The main challenge of live testing is alleviating the risk of test interferences as it may lead to a violation of a system's functional or non-functional requirements. To properly handle this risk, one needs to know which components present a risk of test interferences and what is the cost of the countermeasures to handle that risk. Existing literature relies heavily on human judgement, which can be time consuming, not always feasible, may provide misleading insight. In this paper we go through the challenges of automating this evaluation process and propose a solution to overcome them. Our solution consists of a method for components evaluation which goes through three steps, evaluation of test interferences that may manifest in external behaviour, evaluation of test interferences that may manifest in resource consumption, and finally the evaluation of the cost of implementing the countermeasures to overcome the risk of test interferences.","PeriodicalId":178302,"journal":{"name":"2022 IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW)","volume":"19 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"130949299","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2022-10-01DOI: 10.1109/ISSREW55968.2022.00079
Bahareh Afshinpour, Roland Groz, Massih-Reza Amini
Automated fault identification in long test logs is a tough problem, mainly because of their sequential character and the impossibility of constructing training sets for zero-day faults. To reduce software testers' workload, rule-based approaches have been extensively investigated as solutions for efficiently finding and predicting the fault. Based on software system status monitoring log analysis, we propose a new learning-based technique to automate anomaly detection, correlate test events to anomalies and predict system failures. Since the meaning of fault is not established in system status monitoring-based fault detection, the suggested technique first detects periods of time when a software system status encounters aberrant situations (Bug-Zones). The suggested technique is then tested in a real-time system for anomaly prediction of new tests. The model may be used in two ways. It can assist testers to focus on faulty-like time intervals by reducing the number of test logs. It may also be used to forecast a Bug-Zone in an online system, allowing system administrators to anticipate or even prevent a system failure. An extensive study on a real-world database acquired by a telecommunication operator demonstrates that our approach achieves 71 % accuracy as a Bug-Zones predictor.
{"title":"Correlating Test Events With Monitoring Logs For Test Log Reduction And Anomaly Prediction","authors":"Bahareh Afshinpour, Roland Groz, Massih-Reza Amini","doi":"10.1109/ISSREW55968.2022.00079","DOIUrl":"https://doi.org/10.1109/ISSREW55968.2022.00079","url":null,"abstract":"Automated fault identification in long test logs is a tough problem, mainly because of their sequential character and the impossibility of constructing training sets for zero-day faults. To reduce software testers' workload, rule-based approaches have been extensively investigated as solutions for efficiently finding and predicting the fault. Based on software system status monitoring log analysis, we propose a new learning-based technique to automate anomaly detection, correlate test events to anomalies and predict system failures. Since the meaning of fault is not established in system status monitoring-based fault detection, the suggested technique first detects periods of time when a software system status encounters aberrant situations (Bug-Zones). The suggested technique is then tested in a real-time system for anomaly prediction of new tests. The model may be used in two ways. It can assist testers to focus on faulty-like time intervals by reducing the number of test logs. It may also be used to forecast a Bug-Zone in an online system, allowing system administrators to anticipate or even prevent a system failure. An extensive study on a real-world database acquired by a telecommunication operator demonstrates that our approach achieves 71 % accuracy as a Bug-Zones predictor.","PeriodicalId":178302,"journal":{"name":"2022 IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW)","volume":"33 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"128762729","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
京公网安备 11010802042870号
京ICP备2023020795号-1
扫码关注我们
求助内容：
应助结果提醒方式：