Pub Date : 2020-10-01DOI: 10.1109/ISSREW51248.2020.00081
Lydia Gauerhof, Yuki Hagiwara, Christoph Schorn, M. Trapp
The increased demand of Deep Neural Networks (DNNs) in safety-critical systems, such as autonomous vehicles, leads to increasing importance of training data suitability. Firstly, we focus on how to extract the relevant data content for ensuring DNN reliability. Then, we identify error categories and propose mitigation measures with emphasis on data suitability. Despite all efforts to boost data suitability, not all possible variations of a real application can be identified. Hence, we analyse the case of unknown out-of-distribution data. In this case, we suggest to complement data suitability with online anomaly detection using FACER that supervises the behaviour of the DNN.
{"title":"Considering Reliability of Deep Learning Function to Boost Data Suitability and Anomaly Detection","authors":"Lydia Gauerhof, Yuki Hagiwara, Christoph Schorn, M. Trapp","doi":"10.1109/ISSREW51248.2020.00081","DOIUrl":"https://doi.org/10.1109/ISSREW51248.2020.00081","url":null,"abstract":"The increased demand of Deep Neural Networks (DNNs) in safety-critical systems, such as autonomous vehicles, leads to increasing importance of training data suitability. Firstly, we focus on how to extract the relevant data content for ensuring DNN reliability. Then, we identify error categories and propose mitigation measures with emphasis on data suitability. Despite all efforts to boost data suitability, not all possible variations of a real application can be identified. Hence, we analyse the case of unknown out-of-distribution data. In this case, we suggest to complement data suitability with online anomaly detection using FACER that supervises the behaviour of the DNN.","PeriodicalId":202247,"journal":{"name":"2020 IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW)","volume":"24 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2020-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"124959801","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2020-10-01DOI: 10.1109/ISSREW51248.2020.00032
Miren Illarramendi Rezabal, L. Elorza, F. Larrinaga, Goiuria Sagardui Mendieta
Software Systems are becoming increasingly complex leading to new Validation & Verification challenges. Model checking and testing techniques are used at development time while runtime verification aims to verify that a system satisfies a given property at runtime. This second technique complements the first one. This paper presents a tool that enables the developers to generate automatically reflective UML State Machine controllers and the Runtime Safety Properties Checker (RSPC) which checks a component-based software system’s safety properties defined at design phase. We address embedded systems whose software components are designed by Unified Modelling Language-State Machines (UML-SM) and their internal information can be observed in terms of model elements at runtime. RESCO (REflective State Machines-based observable software COmponents) framework, generates software components that provide this runtime observability. The checker uses software components’ internal status information to check system level safety properties. The checker detects when a system safety property is violated and starts a safe adaptation process to prevent the hazardous scenario. Thus, as demonstrated in the evaluated experiment but not shown in the paper due to the space limitation, the safety of the system is enhanced.
{"title":"CRESCO Framework and Checker: Automatic generation of Reflective UML State Machine’s C++ Code and Checker","authors":"Miren Illarramendi Rezabal, L. Elorza, F. Larrinaga, Goiuria Sagardui Mendieta","doi":"10.1109/ISSREW51248.2020.00032","DOIUrl":"https://doi.org/10.1109/ISSREW51248.2020.00032","url":null,"abstract":"Software Systems are becoming increasingly complex leading to new Validation & Verification challenges. Model checking and testing techniques are used at development time while runtime verification aims to verify that a system satisfies a given property at runtime. This second technique complements the first one. This paper presents a tool that enables the developers to generate automatically reflective UML State Machine controllers and the Runtime Safety Properties Checker (RSPC) which checks a component-based software system’s safety properties defined at design phase. We address embedded systems whose software components are designed by Unified Modelling Language-State Machines (UML-SM) and their internal information can be observed in terms of model elements at runtime. RESCO (REflective State Machines-based observable software COmponents) framework, generates software components that provide this runtime observability. The checker uses software components’ internal status information to check system level safety properties. The checker detects when a system safety property is violated and starts a safe adaptation process to prevent the hazardous scenario. Thus, as demonstrated in the evaluated experiment but not shown in the paper due to the space limitation, the safety of the system is enhanced.","PeriodicalId":202247,"journal":{"name":"2020 IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW)","volume":"19 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2020-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"121345305","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2020-10-01DOI: 10.1109/ISSREW51248.2020.00075
Alessandro Tundo, Chiara Castelnovo, M. Mobilio, O. Riganelli, L. Mariani
Systems of systems are highly dynamic software systems that require flexible monitoring solutions to be observed and controlled. Indeed, operators have to frequently adapt the set of collected indicators according to changing circumstances, to visualize the behavior of the monitored systems and timely take actions, if needed. Unfortunately, dashboard systems are still quite cumbersome to conFigure and adapt to a changing set of indicators that must be visualized.This paper reports our initial effort towards the definition of an automatic dashboard generation process that exploits meta-model layouts to create a full dashboard from a set of indicators selected by operators.
{"title":"Declarative Dashboard Generation","authors":"Alessandro Tundo, Chiara Castelnovo, M. Mobilio, O. Riganelli, L. Mariani","doi":"10.1109/ISSREW51248.2020.00075","DOIUrl":"https://doi.org/10.1109/ISSREW51248.2020.00075","url":null,"abstract":"Systems of systems are highly dynamic software systems that require flexible monitoring solutions to be observed and controlled. Indeed, operators have to frequently adapt the set of collected indicators according to changing circumstances, to visualize the behavior of the monitored systems and timely take actions, if needed. Unfortunately, dashboard systems are still quite cumbersome to conFigure and adapt to a changing set of indicators that must be visualized.This paper reports our initial effort towards the definition of an automatic dashboard generation process that exploits meta-model layouts to create a full dashboard from a set of indicators selected by operators.","PeriodicalId":202247,"journal":{"name":"2020 IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW)","volume":"8 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2020-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"133313605","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2020-10-01DOI: 10.1109/issrew51248.2020.00022
{"title":"Message from the WoSoCer 2020 Workshop Chairs","authors":"","doi":"10.1109/issrew51248.2020.00022","DOIUrl":"https://doi.org/10.1109/issrew51248.2020.00022","url":null,"abstract":"","PeriodicalId":202247,"journal":{"name":"2020 IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW)","volume":"21 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2020-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"129695925","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2020-10-01DOI: 10.1109/ISSREW51248.2020.00089
Johan Sundell, K. Lundqvist, H. Forsberg
Safety-critical software systems have traditionally been found in few domains, e.g., aerospace, nuclear and medical. As technology advances and software capability increases, such systems can be found in more and more applications, e.g., selfdriving cars, autonomous trains. This development will dramatically increase the operational exposure of such systems. All safety-critical applications need to meet exceptionally stringent criteria in terms of dependability. Proving compliance is a challenge for the industry and there is a lack of accepted methods to determine the status of safety-critical software. The regulatory bodies often require a certain amount of testing to be performed but do not, for software systems, require evidence of a given failure rate. This paper addresses quantification of test results. It examines both theoretical and practical aspects. The contribution of this paper is an equation that estimates the remaining undetected faults in the software system after testing. The equation considers partial test coverage. The theoretical results are validated with results from a large industry study (commercial military software). Additionally, the industry results are used to analyze the concept of entropy also known as Shannon information, which is shown to describe the knowledge gained from a test effort.
{"title":"Safety-Critical Software - Quantification of Test Results","authors":"Johan Sundell, K. Lundqvist, H. Forsberg","doi":"10.1109/ISSREW51248.2020.00089","DOIUrl":"https://doi.org/10.1109/ISSREW51248.2020.00089","url":null,"abstract":"Safety-critical software systems have traditionally been found in few domains, e.g., aerospace, nuclear and medical. As technology advances and software capability increases, such systems can be found in more and more applications, e.g., selfdriving cars, autonomous trains. This development will dramatically increase the operational exposure of such systems. All safety-critical applications need to meet exceptionally stringent criteria in terms of dependability. Proving compliance is a challenge for the industry and there is a lack of accepted methods to determine the status of safety-critical software. The regulatory bodies often require a certain amount of testing to be performed but do not, for software systems, require evidence of a given failure rate. This paper addresses quantification of test results. It examines both theoretical and practical aspects. The contribution of this paper is an equation that estimates the remaining undetected faults in the software system after testing. The equation considers partial test coverage. The theoretical results are validated with results from a large industry study (commercial military software). Additionally, the industry results are used to analyze the concept of entropy also known as Shannon information, which is shown to describe the knowledge gained from a test effort.","PeriodicalId":202247,"journal":{"name":"2020 IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW)","volume":"7 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2020-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"130258457","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2020-10-01DOI: 10.1109/ISSREW51248.2020.00046
V. Debroy, Alireza Mansoori, James Haleblian, Mark Wilkens
Application Performance Management (APM) is an important software engineering process that is critical to properly assessing the reliability of software in its targeted environment. Of late, migrating applications to the cloud has become increasingly popular for many reasons such as reduced operational costs, scalability, and increased IT productivity. While much has been said about how to migrate applications, relatively less has been said on how to appropriately monitor them in the cloud, which presents its own unique challenges. This article outlines some of the technical challenges that we have faced at AT&T when migrating to the cloud, with a focus on APM, and aims to stimulate further industrial-academic research and collaboration in this area.
{"title":"Challenges Faced with Application Performance Monitoring (APM) when Migrating to the Cloud","authors":"V. Debroy, Alireza Mansoori, James Haleblian, Mark Wilkens","doi":"10.1109/ISSREW51248.2020.00046","DOIUrl":"https://doi.org/10.1109/ISSREW51248.2020.00046","url":null,"abstract":"Application Performance Management (APM) is an important software engineering process that is critical to properly assessing the reliability of software in its targeted environment. Of late, migrating applications to the cloud has become increasingly popular for many reasons such as reduced operational costs, scalability, and increased IT productivity. While much has been said about how to migrate applications, relatively less has been said on how to appropriately monitor them in the cloud, which presents its own unique challenges. This article outlines some of the technical challenges that we have faced at AT&T when migrating to the cloud, with a focus on APM, and aims to stimulate further industrial-academic research and collaboration in this area.","PeriodicalId":202247,"journal":{"name":"2020 IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW)","volume":"59 19 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2020-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"114882126","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2020-10-01DOI: 10.1109/ISSREW51248.2020.00067
Thomas Hirsch, Birgit Hofer
This paper proposes a supervised machine learning approach for predicting the root cause of a given bug report. Knowing the root cause of a bug can help developers in the debugging process—either directly or indirectly by choosing proper tool support for the debugging task. We mined 54755 closed bug reports from the issue trackers of 103 GitHub projects and applied a set of heuristics to create a benchmark consisting of 10459 reports. A subset was manually classified into three groups (semantic, memory, and concurrency) based on the bugs’ root causes. Since the types of root cause are not equally distributed, a combination of keyword search and random selection was applied. Our data set for the machine learning approach consists of 369 bug reports (122 concurrency, 121 memory, and 126 semantic bugs). The bug reports are used as input to a natural language processing algorithm. We evaluated the performance of several classifiers for predicting the root causes for the given bug reports. Linear Support Vector machines achieved the highest mean precision (0.74) and recall (0.72) scores. The created bug data set and classification are publicly available.
{"title":"Root cause prediction based on bug reports","authors":"Thomas Hirsch, Birgit Hofer","doi":"10.1109/ISSREW51248.2020.00067","DOIUrl":"https://doi.org/10.1109/ISSREW51248.2020.00067","url":null,"abstract":"This paper proposes a supervised machine learning approach for predicting the root cause of a given bug report. Knowing the root cause of a bug can help developers in the debugging process—either directly or indirectly by choosing proper tool support for the debugging task. We mined 54755 closed bug reports from the issue trackers of 103 GitHub projects and applied a set of heuristics to create a benchmark consisting of 10459 reports. A subset was manually classified into three groups (semantic, memory, and concurrency) based on the bugs’ root causes. Since the types of root cause are not equally distributed, a combination of keyword search and random selection was applied. Our data set for the machine learning approach consists of 369 bug reports (122 concurrency, 121 memory, and 126 semantic bugs). The bug reports are used as input to a natural language processing algorithm. We evaluated the performance of several classifiers for predicting the root causes for the given bug reports. Linear Support Vector machines achieved the highest mean precision (0.74) and recall (0.72) scores. The created bug data set and classification are publicly available.","PeriodicalId":202247,"journal":{"name":"2020 IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW)","volume":"1 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2020-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"128557325","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2020-10-01DOI: 10.1109/ISSREW51248.2020.00042
A. Jayakumar, S. Gautham, D. R. Kuhn, B. Simon, Aidan G. Collins, Thomas Dirsch, R. Kacker, C. Elks
While design assurance and testing methods for safety-critical systems have been widely researched and studied for years across a number of industry domains, there are few efforts reported in the literature on the actual application of software testing methods to nuclear power digital I&C systems or devices. We see this as a gap in the knowledge basis. The motivation for this research was to investigate the efficacy and challenges that arise when planning, automating and conducting systematic software testing on actual real-time embedded digital devices. In this paper, we present results on the application of a systematic testing methodology called Pseudo-Exhaustive testing. The systematic testing methods were applied at the unit and module integration levels of the software. The findings suggest that Pseudo Exhaustive testing supported by automated testing technology is an effective approach to testing real-time embedded digital devices in critical nuclear applications.
{"title":"Systematic Software Testing of Critical Embedded Digital Devices in Nuclear Power Applications","authors":"A. Jayakumar, S. Gautham, D. R. Kuhn, B. Simon, Aidan G. Collins, Thomas Dirsch, R. Kacker, C. Elks","doi":"10.1109/ISSREW51248.2020.00042","DOIUrl":"https://doi.org/10.1109/ISSREW51248.2020.00042","url":null,"abstract":"While design assurance and testing methods for safety-critical systems have been widely researched and studied for years across a number of industry domains, there are few efforts reported in the literature on the actual application of software testing methods to nuclear power digital I&C systems or devices. We see this as a gap in the knowledge basis. The motivation for this research was to investigate the efficacy and challenges that arise when planning, automating and conducting systematic software testing on actual real-time embedded digital devices. In this paper, we present results on the application of a systematic testing methodology called Pseudo-Exhaustive testing. The systematic testing methods were applied at the unit and module integration levels of the software. The findings suggest that Pseudo Exhaustive testing supported by automated testing technology is an effective approach to testing real-time embedded digital devices in critical nuclear applications.","PeriodicalId":202247,"journal":{"name":"2020 IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW)","volume":"185 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2020-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"134210053","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2020-10-01DOI: 10.1109/ISSREW51248.2020.00050
Antonio Guerriero
The growing adoption of machine learning (ML) in safety-critical contexts makes reliability evaluation of ML systems a crucial task. Although testing represents one of the most used practices to evaluate the reliability of “traditional” systems, just few techniques can be used to evaluate ML-systems’ reliability due to the oracle problem. In this paper, I present a test oracle surrogate able to automatically classify tests’ outcome to obtain feedback about tests whose expected output is unknown. For this purpose, various sources of knowledge are considered to evaluate the outcome of each test. The aim is to exploit this test oracle surrogate to apply classical testing strategies to perform reliability assessment of ML systems. Some preliminary experiments have been performed considering a Convolutional Neural Network (CNN) and exploiting the well known MNIST dataset. These results promise that the presented technique can be effectively used to evaluate the reliability of ML systems.
{"title":"Reliability Evaluation of ML systems, the oracle problem","authors":"Antonio Guerriero","doi":"10.1109/ISSREW51248.2020.00050","DOIUrl":"https://doi.org/10.1109/ISSREW51248.2020.00050","url":null,"abstract":"The growing adoption of machine learning (ML) in safety-critical contexts makes reliability evaluation of ML systems a crucial task. Although testing represents one of the most used practices to evaluate the reliability of “traditional” systems, just few techniques can be used to evaluate ML-systems’ reliability due to the oracle problem. In this paper, I present a test oracle surrogate able to automatically classify tests’ outcome to obtain feedback about tests whose expected output is unknown. For this purpose, various sources of knowledge are considered to evaluate the outcome of each test. The aim is to exploit this test oracle surrogate to apply classical testing strategies to perform reliability assessment of ML systems. Some preliminary experiments have been performed considering a Convolutional Neural Network (CNN) and exploiting the well known MNIST dataset. These results promise that the presented technique can be effectively used to evaluate the reliability of ML systems.","PeriodicalId":202247,"journal":{"name":"2020 IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW)","volume":"6 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2020-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"125587130","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
京公网安备 11010802042870号
京ICP备2023020795号-1
扫码关注我们
求助内容：
应助结果提醒方式：