Pub Date : 2020-10-01DOI: 10.1109/ISSREW51248.2020.00085
Sylvaine Picard, Camille Chapdelaine, Cyril Cappi, L. Gardes, E. Jenn, Baptiste Lefèvre, Thomas Soumarmon
In this paper, we address the problem of dataset quality in the context of Machine Learning (ML)-based critical systems. We briefly analyse the applicability of some existing standards dealing with data and show that the specificities of the ML context are neither properly captured nor taken into account. As a first answer to this concerning situation, we propose a dataset specification and verification process, and apply it on a signal recognition system from the railway domain. In addition, we also give a list of recommendations for the collection and management of datasets. This work is one step towards the dataset engineering process that will be required for ML to be used on safety critical systems.
{"title":"Ensuring Dataset Quality for Machine Learning Certification","authors":"Sylvaine Picard, Camille Chapdelaine, Cyril Cappi, L. Gardes, E. Jenn, Baptiste Lefèvre, Thomas Soumarmon","doi":"10.1109/ISSREW51248.2020.00085","DOIUrl":"https://doi.org/10.1109/ISSREW51248.2020.00085","url":null,"abstract":"In this paper, we address the problem of dataset quality in the context of Machine Learning (ML)-based critical systems. We briefly analyse the applicability of some existing standards dealing with data and show that the specificities of the ML context are neither properly captured nor taken into account. As a first answer to this concerning situation, we propose a dataset specification and verification process, and apply it on a signal recognition system from the railway domain. In addition, we also give a list of recommendations for the collection and management of datasets. This work is one step towards the dataset engineering process that will be required for ML to be used on safety critical systems.","PeriodicalId":202247,"journal":{"name":"2020 IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW)","volume":"6 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2020-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"133635802","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2020-10-01DOI: 10.1109/ISSREW51248.2020.00055
Jinyi Zhou, Kun Qiu, Zheng Zheng, T. Chen, P. Poon
Generating test cases and further evaluating their “quality” are two critical topics in the area of Deep Neural Networks (DNNs). In this domain, different studies (e.g., [1], [2]) have reported that metamorphic testing (MT) serves as an effective test case generation method, where an initial set of source test cases is augmented with identified metamorphic relations (MRs) to produce the corresponding set of follow-up test cases. As a result, the fault detection effectiveness (and, hence, the “quality”) of the resulting test suite T, containing these source and follow-up test cases, will most likely be increased.
{"title":"Using Metamorphic Testing to Evaluate DNN Coverage Criteria","authors":"Jinyi Zhou, Kun Qiu, Zheng Zheng, T. Chen, P. Poon","doi":"10.1109/ISSREW51248.2020.00055","DOIUrl":"https://doi.org/10.1109/ISSREW51248.2020.00055","url":null,"abstract":"Generating test cases and further evaluating their “quality” are two critical topics in the area of Deep Neural Networks (DNNs). In this domain, different studies (e.g., [1], [2]) have reported that metamorphic testing (MT) serves as an effective test case generation method, where an initial set of source test cases is augmented with identified metamorphic relations (MRs) to produce the corresponding set of follow-up test cases. As a result, the fault detection effectiveness (and, hence, the “quality”) of the resulting test suite T, containing these source and follow-up test cases, will most likely be increased.","PeriodicalId":202247,"journal":{"name":"2020 IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW)","volume":"80 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2020-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"126203972","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2020-10-01DOI: 10.1109/ISSREW51248.2020.00076
B. Wudka, Carsten Thomas, Lennart Siefke, V. Sommer
The drive for digitalization in industry and transport results in an increasing application of cooperative systems that form adaptive system-of-systems. With reconfiguration these systems are able to change their behavior to react on internal changes and changes in their environment. In this paper, we present a novel service-oriented approach for decentralized reconfiguration within such systems-of-systems that specifically supports open adaptive systems-of-systems. We introduce the concept of strategy blueprints that define possible combinations of services provided by individual members of the system-of-systems. During reconfiguration, individual member systems evaluate all strategies that are instantiable under the given conditions, and select the one optimally fulfilling a set of predefined goals as the most appropriate target configuration. With this novel approach, we support flexible reconfiguration across the borders of individual member systems, and the inclusion of new member system types that provide service variants not yet known at design time of the reconfiguration algorithm.
{"title":"A Reconfiguration Approach for Open Adaptive Systems-of-Systems","authors":"B. Wudka, Carsten Thomas, Lennart Siefke, V. Sommer","doi":"10.1109/ISSREW51248.2020.00076","DOIUrl":"https://doi.org/10.1109/ISSREW51248.2020.00076","url":null,"abstract":"The drive for digitalization in industry and transport results in an increasing application of cooperative systems that form adaptive system-of-systems. With reconfiguration these systems are able to change their behavior to react on internal changes and changes in their environment. In this paper, we present a novel service-oriented approach for decentralized reconfiguration within such systems-of-systems that specifically supports open adaptive systems-of-systems. We introduce the concept of strategy blueprints that define possible combinations of services provided by individual members of the system-of-systems. During reconfiguration, individual member systems evaluate all strategies that are instantiable under the given conditions, and select the one optimally fulfilling a set of predefined goals as the most appropriate target configuration. With this novel approach, we support flexible reconfiguration across the borders of individual member systems, and the inclusion of new member system types that provide service variants not yet known at design time of the reconfiguration algorithm.","PeriodicalId":202247,"journal":{"name":"2020 IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW)","volume":"40 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2020-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"126846821","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2020-10-01DOI: 10.1109/ISSREW51248.2020.00082
Giacomo Iadarola, F. Martinelli, F. Mercaldo, A. Santone
Artificial intelligence techniques are nowadays widespread to perform a great number of classification tasks. One of the biggest controversies regarding the adoption of these techniques is related to their use as a “black box” i.e., the security analyst must trust the prediction without the possibility to understand the reason why the classifier made a certain choice. In this paper we propose a malicious family detector based on deep learning, providing a mechanism aimed to assess the prediction reliability. The proposed method obtains an accuracy of 0.98 in Android family identification. Moreover, we show how the proposed method can assist the security analyst to interpret the output classification and verify the prediction reliability by exploiting activation maps.
{"title":"Evaluating Deep Learning Classification Reliability in Android Malware Family Detection","authors":"Giacomo Iadarola, F. Martinelli, F. Mercaldo, A. Santone","doi":"10.1109/ISSREW51248.2020.00082","DOIUrl":"https://doi.org/10.1109/ISSREW51248.2020.00082","url":null,"abstract":"Artificial intelligence techniques are nowadays widespread to perform a great number of classification tasks. One of the biggest controversies regarding the adoption of these techniques is related to their use as a “black box” i.e., the security analyst must trust the prediction without the possibility to understand the reason why the classifier made a certain choice. In this paper we propose a malicious family detector based on deep learning, providing a mechanism aimed to assess the prediction reliability. The proposed method obtains an accuracy of 0.98 in Android family identification. Moreover, we show how the proposed method can assist the security analyst to interpret the output classification and verify the prediction reliability by exploiting activation maps.","PeriodicalId":202247,"journal":{"name":"2020 IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW)","volume":"8 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2020-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"128989886","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2020-10-01DOI: 10.1109/ISSREW51248.2020.00098
Junjun Zheng, H. Okamura, T. Dohi
Robustness is usually relevant for characterizing the dependence between the values of model parameters and system behavior, and can be understood as stability of system behavior under changes in model parameters. In this paper, we consider a simple software rejuvenation model and its optimal rejuvenation timing, which maximizes the steady-state availability of the system. The main contribution of this work is to provide a new perspective on the optimal software rejuvenation timing, that is, the robustness of the optimal rejuvenation timing against input factors. In particular, the degree of robustness is quantified by the first derivatives of the optimal rejuvenation timing with respect to the model parameters. The robustnesses of both optimal rejuvenation timing and system availability with the optimal rejuvenation timing are considered. A numerical example with Weibull distributed failure time is devoted to clarifying how robust the optimal rejuvenation timing is, and determine the most sensitive model parameter. As a result, the optimal rejuvenation timing seems to be more robust to the parameters regarding failure time distribution, compared with the other parameters.
{"title":"How Robust is the Optimal Software Rejuvenation Timing?","authors":"Junjun Zheng, H. Okamura, T. Dohi","doi":"10.1109/ISSREW51248.2020.00098","DOIUrl":"https://doi.org/10.1109/ISSREW51248.2020.00098","url":null,"abstract":"Robustness is usually relevant for characterizing the dependence between the values of model parameters and system behavior, and can be understood as stability of system behavior under changes in model parameters. In this paper, we consider a simple software rejuvenation model and its optimal rejuvenation timing, which maximizes the steady-state availability of the system. The main contribution of this work is to provide a new perspective on the optimal software rejuvenation timing, that is, the robustness of the optimal rejuvenation timing against input factors. In particular, the degree of robustness is quantified by the first derivatives of the optimal rejuvenation timing with respect to the model parameters. The robustnesses of both optimal rejuvenation timing and system availability with the optimal rejuvenation timing are considered. A numerical example with Weibull distributed failure time is devoted to clarifying how robust the optimal rejuvenation timing is, and determine the most sensitive model parameter. As a result, the optimal rejuvenation timing seems to be more robust to the parameters regarding failure time distribution, compared with the other parameters.","PeriodicalId":202247,"journal":{"name":"2020 IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW)","volume":"25 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2020-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"125465908","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2020-10-01DOI: 10.1109/ISSREW51248.2020.00078
Charles F. Gonçalves, Nuno Antunes
Hypervisors govern the resources of virtualized systems and are a crucial component of many cloud solutions. As a critical component, cloud providers should assess the hypervisor’s security to mitigate risk before adoption. Ideally, a benchmark should be applied to compare the security of different systems objectively, but security benchmarking is still an open problem. Notwithstanding, the evaluation of the system’s trustworthiness has been adopted as a promising approach as part of this complex evaluation process. In this work, we present a vulnerability data analysis of the Xen hypervisor. Additionally, we address the problem of how to apply this analysis results as trustworthiness evidence that can be applied in security benchmarks. Our results present an insightful characterization of Xen’s vulnerabilities evaluating their lifespan, distribution, and modeling. We also show that vulnerability data analysis can qualitatively characterize the Xen hypervisor’s trustworthiness and possibly reflect the security development efforts into its codebase.
{"title":"Vulnerability Analysis as Trustworthiness Evidence in Security Benchmarking: A Case Study on Xen.","authors":"Charles F. Gonçalves, Nuno Antunes","doi":"10.1109/ISSREW51248.2020.00078","DOIUrl":"https://doi.org/10.1109/ISSREW51248.2020.00078","url":null,"abstract":"Hypervisors govern the resources of virtualized systems and are a crucial component of many cloud solutions. As a critical component, cloud providers should assess the hypervisor’s security to mitigate risk before adoption. Ideally, a benchmark should be applied to compare the security of different systems objectively, but security benchmarking is still an open problem. Notwithstanding, the evaluation of the system’s trustworthiness has been adopted as a promising approach as part of this complex evaluation process. In this work, we present a vulnerability data analysis of the Xen hypervisor. Additionally, we address the problem of how to apply this analysis results as trustworthiness evidence that can be applied in security benchmarks. Our results present an insightful characterization of Xen’s vulnerabilities evaluating their lifespan, distribution, and modeling. We also show that vulnerability data analysis can qualitatively characterize the Xen hypervisor’s trustworthiness and possibly reflect the security development efforts into its codebase.","PeriodicalId":202247,"journal":{"name":"2020 IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW)","volume":"15 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2020-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"130604440","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2020-10-01DOI: 10.1109/ISSREW51248.2020.00034
Muhammad Usman Sarwar, Sarim Zafar, Mohamed Wiem Mkaouer, G. Walia, Muhammad Zubair Malik
Commit messages are used in the industry by developers to annotate changes made to the code. Accurate classification of these messages can help monitor the software evolution process and enable better tracking for various industrial stakeholders. In this paper, we present a state of the art method for commit message classification into categories as per Swanson’s maintenance activities i.e. “Corrective”, “Perfective”, and “Adaptive”. This is a challenging task because not all commit messages are well written and informative. Existing approaches rely on keyword-based techniques to solve this problem. However, these approaches are oblivious to the full language model and do not recognize the contextual relationship between words. State of the art methodology in Natural Language Processing (NLP), is to train a context-aware neural network (Transformer) on a very large data set that encompasses the entire language and then fine-tunes it for a specific task. In this way, the model can learn the language, pay attention to the context, and then transfer that knowledge for better performance at the specific task. We use an off-the-shelf neural network called DistilBERT and fine-tune it for commit message classification task. This step is non-trivial because programming languages and commit messages have unique keywords, jargon, and idioms. This paper presents our effort in training this model and constructing the data set for this task. We describe the rules used to construct the data set. We validate our approach on industrial projects from GitHub, such as Kubernetes, Linux, TensorFlow, Spark, TypeScript, and PyTorch. We were able to achieve 87% F1-score for the commit message classification task, which is an order of magnitude accurate than previous studies.
{"title":"Multi-label Classification of Commit Messages using Transfer Learning","authors":"Muhammad Usman Sarwar, Sarim Zafar, Mohamed Wiem Mkaouer, G. Walia, Muhammad Zubair Malik","doi":"10.1109/ISSREW51248.2020.00034","DOIUrl":"https://doi.org/10.1109/ISSREW51248.2020.00034","url":null,"abstract":"Commit messages are used in the industry by developers to annotate changes made to the code. Accurate classification of these messages can help monitor the software evolution process and enable better tracking for various industrial stakeholders. In this paper, we present a state of the art method for commit message classification into categories as per Swanson’s maintenance activities i.e. “Corrective”, “Perfective”, and “Adaptive”. This is a challenging task because not all commit messages are well written and informative. Existing approaches rely on keyword-based techniques to solve this problem. However, these approaches are oblivious to the full language model and do not recognize the contextual relationship between words. State of the art methodology in Natural Language Processing (NLP), is to train a context-aware neural network (Transformer) on a very large data set that encompasses the entire language and then fine-tunes it for a specific task. In this way, the model can learn the language, pay attention to the context, and then transfer that knowledge for better performance at the specific task. We use an off-the-shelf neural network called DistilBERT and fine-tune it for commit message classification task. This step is non-trivial because programming languages and commit messages have unique keywords, jargon, and idioms. This paper presents our effort in training this model and constructing the data set for this task. We describe the rules used to construct the data set. We validate our approach on industrial projects from GitHub, such as Kubernetes, Linux, TensorFlow, Spark, TypeScript, and PyTorch. We were able to achieve 87% F1-score for the commit message classification task, which is an order of magnitude accurate than previous studies.","PeriodicalId":202247,"journal":{"name":"2020 IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW)","volume":"21 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2020-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"133961935","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2020-10-01DOI: 10.1109/issrew51248.2020.00013
Pietro Braione, D. Briola, G. Angelis, F. Gallo, F. Poggi, G. Quattrocchi
This year too, GAUSS has been co-located with the IEEE “International Symposium on Software Reliability Engineering (ISSRE)”. We would thank the organizers of ISSRE 2020 for having hosted our Workshop, and the ISSRE workshop chairs for their support during the organization of this 2 edition. A very special thank goes to all the members of the program committee, eighteen specialists, for their effort and professional reviews.
{"title":"Message from the GAUSS 2020 Workshop Chairs","authors":"Pietro Braione, D. Briola, G. Angelis, F. Gallo, F. Poggi, G. Quattrocchi","doi":"10.1109/issrew51248.2020.00013","DOIUrl":"https://doi.org/10.1109/issrew51248.2020.00013","url":null,"abstract":"This year too, GAUSS has been co-located with the IEEE “International Symposium on Software Reliability Engineering (ISSRE)”. We would thank the organizers of ISSRE 2020 for having hosted our Workshop, and the ISSRE workshop chairs for their support during the organization of this 2 edition. A very special thank goes to all the members of the program committee, eighteen specialists, for their effort and professional reviews.","PeriodicalId":202247,"journal":{"name":"2020 IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW)","volume":"401 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2020-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"132208183","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2020-10-01DOI: 10.1109/ISSREW51248.2020.00071
Matteo Camilli, Carmine Colarusso, B. Russo, E. Zimeo
The microservices architectural style is picking up more and more momentum in IT industry for the development of systems as loosely coupled, collaborating services. Companies that undergo the migration of their own applications have aspirations such as increasing maintainability and the scale of operation. Such a process is worthwhile but not easy, since it should ensure atomic improvements to the overall architecture for each migration step. Furthermore, the systematic evaluation of migration steps becomes cumbersome without sensible optimization metrics that take into account performance and scalability under expected operational conditions. Recent lines of research recognize this task as challenging, especially in data-intensive applications where known approaches based, for instance, on Domain Driven Design may not be adequate. In this paper, we introduce an approach to evaluate a migration in an iterative way and recognize whether it represents an improvement in terms of performance and scalability. The approach leverages a Domain Metric-based analysis to quantitatively evaluate alternative architectures. We exemplified the envisioned approach on a data-intensive application case study in the domain of smart mobility. Preliminary results from our controlled experiments show the effectiveness of our approach to support systematic and automated evaluation of migration processes.
{"title":"Domain Metric Driven Decomposition of Data-Intensive Applications","authors":"Matteo Camilli, Carmine Colarusso, B. Russo, E. Zimeo","doi":"10.1109/ISSREW51248.2020.00071","DOIUrl":"https://doi.org/10.1109/ISSREW51248.2020.00071","url":null,"abstract":"The microservices architectural style is picking up more and more momentum in IT industry for the development of systems as loosely coupled, collaborating services. Companies that undergo the migration of their own applications have aspirations such as increasing maintainability and the scale of operation. Such a process is worthwhile but not easy, since it should ensure atomic improvements to the overall architecture for each migration step. Furthermore, the systematic evaluation of migration steps becomes cumbersome without sensible optimization metrics that take into account performance and scalability under expected operational conditions. Recent lines of research recognize this task as challenging, especially in data-intensive applications where known approaches based, for instance, on Domain Driven Design may not be adequate. In this paper, we introduce an approach to evaluate a migration in an iterative way and recognize whether it represents an improvement in terms of performance and scalability. The approach leverages a Domain Metric-based analysis to quantitatively evaluate alternative architectures. We exemplified the envisioned approach on a data-intensive application case study in the domain of smart mobility. Preliminary results from our controlled experiments show the effectiveness of our approach to support systematic and automated evaluation of migration processes.","PeriodicalId":202247,"journal":{"name":"2020 IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW)","volume":"27 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2020-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"122218696","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
京公网安备 11010802042870号
京ICP备2023020795号-1
扫码关注我们
求助内容：
应助结果提醒方式：