Pub Date : 2022-04-01DOI: 10.1109/ICSTW55395.2022.00049
Yasuaki Hiruta, Hidetoshi Suhara, Y. Nishi
Model-based Testing (MBT), is an important testing technology to generate test cases automatically by MBT tools. While a single MBT tool supports a single MBT model, it is necessary to integrate implicitly multiple models to detect failures manually such as exploratory testing in industry. A single-model-MBT lacks fidelity which means the ability to generate the required test cases. In this research, we analyse such failures to identify implicit multiple models. Several patterns on integration of multiple models are discovered. This paper introduces the patterns and examples of test case generation to improve fidelity.
{"title":"Patterns to Improve Fidelity for Model-Based Testing","authors":"Yasuaki Hiruta, Hidetoshi Suhara, Y. Nishi","doi":"10.1109/ICSTW55395.2022.00049","DOIUrl":"https://doi.org/10.1109/ICSTW55395.2022.00049","url":null,"abstract":"Model-based Testing (MBT), is an important testing technology to generate test cases automatically by MBT tools. While a single MBT tool supports a single MBT model, it is necessary to integrate implicitly multiple models to detect failures manually such as exploratory testing in industry. A single-model-MBT lacks fidelity which means the ability to generate the required test cases. In this research, we analyse such failures to identify implicit multiple models. Several patterns on integration of multiple models are discovered. This paper introduces the patterns and examples of test case generation to improve fidelity.","PeriodicalId":147133,"journal":{"name":"2022 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)","volume":"350 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-04-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"123328887","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2022-04-01DOI: 10.1109/ICSTW55395.2022.00046
Marko Savić, M. Mäntylä, Maëlick Claes
Despite the widespread of test automation, automatic testing of graphical user interfaces (GUI) remains a challenge. This is partly due to the difficulty of reliably identifying GUI elements over different versions of a given software system. Machine vision techniques could be a potential way of addressing this issue by automatically identifying GUI elements with the help of machine learning. However, developing a GUI testing tool relying on automatic identification of graphical elements first requires to acquire large amount of labeled data. In this paper, we present Win GUI Crawler, a tool for automatically gathering such data from Microsoft Windows GUI applications. The tool is based on Microsoft Windows Application Driver and performs actions on the GUI using a depth-first traversal of the GUI element tree. For each action performed by the crawler, screenshots are taken and metadata is extracted for each of the different screens. Bounding boxes of GUI elements are then filtered in order to identify what GUI elements are actually visible on the screen. Win GUI Crawler is then evaluated on several popular Windows applications and the current limitations are discussed.
尽管自动化测试的普及，图形用户界面(GUI)的自动化测试仍然是一个挑战。这部分是由于难以在给定软件系统的不同版本上可靠地识别GUI元素。机器视觉技术可以通过在机器学习的帮助下自动识别GUI元素来解决这个问题。然而，开发依赖于图形元素自动识别的GUI测试工具首先需要获取大量的标记数据。在本文中，我们介绍了Win GUI Crawler，一个从Microsoft Windows GUI应用程序自动收集此类数据的工具。该工具基于Microsoft Windows Application Driver，并使用GUI元素树的深度优先遍历在GUI上执行操作。对于爬虫执行的每个操作，都会截取屏幕截图，并为每个不同的屏幕提取元数据。然后过滤GUI元素的边界框，以确定哪些GUI元素在屏幕上实际可见。然后在几种流行的Windows应用程序上对Win GUI Crawler进行了评估，并讨论了当前的限制。
{"title":"Win GUI Crawler: A tool prototype for desktop GUI image and metadata collection","authors":"Marko Savić, M. Mäntylä, Maëlick Claes","doi":"10.1109/ICSTW55395.2022.00046","DOIUrl":"https://doi.org/10.1109/ICSTW55395.2022.00046","url":null,"abstract":"Despite the widespread of test automation, automatic testing of graphical user interfaces (GUI) remains a challenge. This is partly due to the difficulty of reliably identifying GUI elements over different versions of a given software system. Machine vision techniques could be a potential way of addressing this issue by automatically identifying GUI elements with the help of machine learning. However, developing a GUI testing tool relying on automatic identification of graphical elements first requires to acquire large amount of labeled data. In this paper, we present Win GUI Crawler, a tool for automatically gathering such data from Microsoft Windows GUI applications. The tool is based on Microsoft Windows Application Driver and performs actions on the GUI using a depth-first traversal of the GUI element tree. For each action performed by the crawler, screenshots are taken and metadata is extracted for each of the different screens. Bounding boxes of GUI elements are then filtered in order to identify what GUI elements are actually visible on the screen. Win GUI Crawler is then evaluated on several popular Windows applications and the current limitations are discussed.","PeriodicalId":147133,"journal":{"name":"2022 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)","volume":"157 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-04-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"122920165","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2022-04-01DOI: 10.1109/ICSTW55395.2022.00030
A. Patel, Jaganmohan Chandrasekaran, Yu Lei, R. Kacker, D. R. Kuhn
Machine Learning (ML) models could exhibit biased behavior, or algorithmic discrimination, resulting in unfair or discriminatory outcomes. The bias in the ML model could emanate from various factors such as the training dataset, the choice of the ML algorithm, or the hyperparameters used to train the ML model. In addition to evaluating the model’s correctness, it is essential to test ML models for fair and unbiased behavior. In this paper, we present a combinatorial testing-based approach to perform fairness testing of ML models. Our approach is model agnostic and evaluates fairness violations of a pre-trained ML model in a two-step process. In the first step, we create an input parameter model from the training data set and then use the model to generate a t-way test set. In the second step, for each test, we modify the value of one or more protected attributes to see if we could find fairness violations. We performed an experimental evaluation of the proposed approach using ML models trained with tabular datasets. The results suggest that the proposed approach can successfully identify fairness violations in pre-trained ML models.
{"title":"A Combinatorial Approach to Fairness Testing of Machine Learning Models","authors":"A. Patel, Jaganmohan Chandrasekaran, Yu Lei, R. Kacker, D. R. Kuhn","doi":"10.1109/ICSTW55395.2022.00030","DOIUrl":"https://doi.org/10.1109/ICSTW55395.2022.00030","url":null,"abstract":"Machine Learning (ML) models could exhibit biased behavior, or algorithmic discrimination, resulting in unfair or discriminatory outcomes. The bias in the ML model could emanate from various factors such as the training dataset, the choice of the ML algorithm, or the hyperparameters used to train the ML model. In addition to evaluating the model’s correctness, it is essential to test ML models for fair and unbiased behavior. In this paper, we present a combinatorial testing-based approach to perform fairness testing of ML models. Our approach is model agnostic and evaluates fairness violations of a pre-trained ML model in a two-step process. In the first step, we create an input parameter model from the training data set and then use the model to generate a t-way test set. In the second step, for each test, we modify the value of one or more protected attributes to see if we could find fairness violations. We performed an experimental evaluation of the proposed approach using ML models trained with tabular datasets. The results suggest that the proposed approach can successfully identify fairness violations in pre-trained ML models.","PeriodicalId":147133,"journal":{"name":"2022 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)","volume":"37 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-04-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"124704715","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2022-04-01DOI: 10.1109/ICSTW55395.2022.00047
Tomohiro Takeda, Satoshi Masuda, K. Tsuda
Quality assurance is one of the most important activities in software development and maintenance. Software source codes are modified via change requests, functional improvement, and refactoring. When software changes, it is difficult to define the scope of test cases, and software testing costs tend to increase to maintain software quality. Therefore, change analysis is a challenge, and static testing is a key solution to this challenge. In this study, we propose new static testing metrics using mathematical graph analysis techniques for the control flow graph generated from the three-address code of the implementation codes based on our hypothesis of the existing correlation between the graph features and any software bugs. Five graph features are strongly correlated with the software bugs. Hence, our bug prediction model exhibits a better performance of 0.25 FN, 0.04 TN ratio, and 0.08 precision than a model based on the traditional bug prediction metrics, which are complexity, line of code (steps), and CRUD.
{"title":"Software Bug Prediction Model Based on Mathematical Graph Features Metrics","authors":"Tomohiro Takeda, Satoshi Masuda, K. Tsuda","doi":"10.1109/ICSTW55395.2022.00047","DOIUrl":"https://doi.org/10.1109/ICSTW55395.2022.00047","url":null,"abstract":"Quality assurance is one of the most important activities in software development and maintenance. Software source codes are modified via change requests, functional improvement, and refactoring. When software changes, it is difficult to define the scope of test cases, and software testing costs tend to increase to maintain software quality. Therefore, change analysis is a challenge, and static testing is a key solution to this challenge. In this study, we propose new static testing metrics using mathematical graph analysis techniques for the control flow graph generated from the three-address code of the implementation codes based on our hypothesis of the existing correlation between the graph features and any software bugs. Five graph features are strongly correlated with the software bugs. Hence, our bug prediction model exhibits a better performance of 0.25 FN, 0.04 TN ratio, and 0.08 precision than a model based on the traditional bug prediction metrics, which are complexity, line of code (steps), and CRUD.","PeriodicalId":147133,"journal":{"name":"2022 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)","volume":"2 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-04-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"115244055","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2022-04-01DOI: 10.1109/ICSTW55395.2022.00025
Naoko Okubo, Shoma Takatsuki, Yasushi Ueda
The functionality of Fault Detection, Isolation, and Recovery (FDIR) is a key factor in achieving the high reliability of space systems. The test suites for the FDIR functionality in JAXA’s space systems are manually designed by expert engineers with decades of experience to achieve as high combination coverage with a small test suite as possible. However, there are only a few engineers who can perform such ad-hoc test suite design. Therefore, FDIR functionality testing requires a supportive method to generate a test suite with the high combination coverage with the smallest size that can be executed in the development timescale. In this paper, we describe our experience in applying popular combinatorial testing techniques to generate the real-world earth-observation satellite’s FDIR functionality test suites and comparing them with conventional human-derived test suite. The purpose of this comparison is to check the capability of the existing combinatorial testing methods toward FDIR functionality testing. Here, the FDIR functionality testing were treated as combinatorial configuration testing. As a result, we found that the 2-way coverage rate by the human, PICT, ACTS and the HAYST method were 72.7%, 66.3%, 68.8% and 72.2% with 16, 10, 10 and 14 test cases, respectively.
{"title":"Experience of Combinatorial Testing toward Fault Detection, Isolation and Recovery Functionality","authors":"Naoko Okubo, Shoma Takatsuki, Yasushi Ueda","doi":"10.1109/ICSTW55395.2022.00025","DOIUrl":"https://doi.org/10.1109/ICSTW55395.2022.00025","url":null,"abstract":"The functionality of Fault Detection, Isolation, and Recovery (FDIR) is a key factor in achieving the high reliability of space systems. The test suites for the FDIR functionality in JAXA’s space systems are manually designed by expert engineers with decades of experience to achieve as high combination coverage with a small test suite as possible. However, there are only a few engineers who can perform such ad-hoc test suite design. Therefore, FDIR functionality testing requires a supportive method to generate a test suite with the high combination coverage with the smallest size that can be executed in the development timescale. In this paper, we describe our experience in applying popular combinatorial testing techniques to generate the real-world earth-observation satellite’s FDIR functionality test suites and comparing them with conventional human-derived test suite. The purpose of this comparison is to check the capability of the existing combinatorial testing methods toward FDIR functionality testing. Here, the FDIR functionality testing were treated as combinatorial configuration testing. As a result, we found that the 2-way coverage rate by the human, PICT, ACTS and the HAYST method were 72.7%, 66.3%, 68.8% and 72.2% with 16, 10, 10 and 14 test cases, respectively.","PeriodicalId":147133,"journal":{"name":"2022 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)","volume":"30 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-04-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"126481602","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2022-04-01DOI: 10.1109/ICSTW55395.2022.00032
D. Kuhn, M. Raunak, Charles B. Prado, Vinay C. Patil, R. Kacker
Combinatorial coverage measures have been defined and applied to a wide range of problems. These methods have been developed using measures that depend on the inclusion or absence of t-tuples of values in inputs and test cases. We extend these coverage measures to include the frequency of occurrence of combinations, in an approach that we refer to as combination frequency differencing (CFD). This method is particularly suited to artificial intelligence and machine learning (AI/ML) applications, where training data sets used in learning systems are dependent on the prevalence of various attributes of elements of class and non-class sets. We illustrate the use of this method by applying it to analyzing the susceptibility of physical unclonable functions (PUFs) to machine learning attacks. Preliminary results suggest that the method may be useful for identifying bit combinations that have a disproportionately strong influence on PUF response bit values.
{"title":"Combination Frequency Differencing for Identifying Design Weaknesses in Physical Unclonable Functions","authors":"D. Kuhn, M. Raunak, Charles B. Prado, Vinay C. Patil, R. Kacker","doi":"10.1109/ICSTW55395.2022.00032","DOIUrl":"https://doi.org/10.1109/ICSTW55395.2022.00032","url":null,"abstract":"Combinatorial coverage measures have been defined and applied to a wide range of problems. These methods have been developed using measures that depend on the inclusion or absence of t-tuples of values in inputs and test cases. We extend these coverage measures to include the frequency of occurrence of combinations, in an approach that we refer to as combination frequency differencing (CFD). This method is particularly suited to artificial intelligence and machine learning (AI/ML) applications, where training data sets used in learning systems are dependent on the prevalence of various attributes of elements of class and non-class sets. We illustrate the use of this method by applying it to analyzing the susceptibility of physical unclonable functions (PUFs) to machine learning attacks. Preliminary results suggest that the method may be useful for identifying bit combinations that have a disproportionately strong influence on PUF response bit values.","PeriodicalId":147133,"journal":{"name":"2022 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)","volume":"27 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-04-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"124453473","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2022-04-01DOI: 10.1109/ICSTW55395.2022.00015
Ildar Nigmatullin, A. Sadovykh, Nan Messe, S. Ebersold, J. Bruel
For the last 20 years, the number of vulnerabilities has increased near 20 times, according to NIST statistics. Vulnerabilities expose companies to risks that may seriously threaten their operations. Therefore, for a long time, it has been suggested to apply security engineering – the process of accumulating multiple techniques and practices to ensure a sufficient level of security and to prevent vulnerabilities in the early stages of software development, including establishing security requirements and proper security testing. The informal nature of security requirements makes it uneasy to maintain system security, eliminate redundancy and trace requirements down to verification artifacts such as test cases. To deal with this problem, Seamless Object-Oriented Requirements (SOORs) promote incorporating formal requirements representations and verification means together into requirements classes.This article is a position paper that discusses opportunities to implement the Requirements as Code (RQCODE) concepts, SOORs in Java, applied to the Software Security domain. We argue that this concept has an elegance and the potential to raise the attention of developers since it combines a lightweight formalization of requirements through security tests with seamless integration with off-the-shelf development environments, including modern Continuous Integration/Delivery platforms. The benefits of this approach are yet to be demonstrated in further studies in the VeriDevOps project.
{"title":"RQCODE – Towards Object-Oriented Requirements in the Software Security Domain","authors":"Ildar Nigmatullin, A. Sadovykh, Nan Messe, S. Ebersold, J. Bruel","doi":"10.1109/ICSTW55395.2022.00015","DOIUrl":"https://doi.org/10.1109/ICSTW55395.2022.00015","url":null,"abstract":"For the last 20 years, the number of vulnerabilities has increased near 20 times, according to NIST statistics. Vulnerabilities expose companies to risks that may seriously threaten their operations. Therefore, for a long time, it has been suggested to apply security engineering – the process of accumulating multiple techniques and practices to ensure a sufficient level of security and to prevent vulnerabilities in the early stages of software development, including establishing security requirements and proper security testing. The informal nature of security requirements makes it uneasy to maintain system security, eliminate redundancy and trace requirements down to verification artifacts such as test cases. To deal with this problem, Seamless Object-Oriented Requirements (SOORs) promote incorporating formal requirements representations and verification means together into requirements classes.This article is a position paper that discusses opportunities to implement the Requirements as Code (RQCODE) concepts, SOORs in Java, applied to the Software Security domain. We argue that this concept has an elegance and the potential to raise the attention of developers since it combines a lightweight formalization of requirements through security tests with seamless integration with off-the-shelf development environments, including modern Continuous Integration/Delivery platforms. The benefits of this approach are yet to be demonstrated in further studies in the VeriDevOps project.","PeriodicalId":147133,"journal":{"name":"2022 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)","volume":"19 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-04-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"126520747","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2022-04-01DOI: 10.1109/ICSTW55395.2022.00057
Azeem Ahmad, Erik Norrestam Held, O. Leifler, K. Sandahl
Developers often spend time to determine whether test case failures are real failures or flaky. The flaky tests, known as non-deterministic tests, change their outcomes without any changes in the codebase, thus reducing the trust of developers during a software release as well as in the quality of a product. While rerunning test cases is a common approach, it is resource intensive, unreliable, and does not uncover the actual cause of test flakiness. Our paper evaluates an approach to identify randomness-related flaky. This paper used a divergence algorithm and execution tracing techniques to identify flaky tests, which resulted in the FlakyPy prototype. In addition, this paper discusses the cases where FlakyPy successfully identified the flaky test as well as those cases where FlakyPy failed. The papers discuss how the reporting mechanism of FlakyPy can help developers in identifying the root cause of randomness-related test flakiness. Thirty-two open-source projects were used in this. We concluded that FlakyPy can detect most of the randomness-related test flakiness. In addition, the reporting mechanism of FlakyPy reveals sufficient information about possible root causes of test flakiness.
{"title":"Identifying Randomness related Flaky Tests through Divergence and Execution Tracing","authors":"Azeem Ahmad, Erik Norrestam Held, O. Leifler, K. Sandahl","doi":"10.1109/ICSTW55395.2022.00057","DOIUrl":"https://doi.org/10.1109/ICSTW55395.2022.00057","url":null,"abstract":"Developers often spend time to determine whether test case failures are real failures or flaky. The flaky tests, known as non-deterministic tests, change their outcomes without any changes in the codebase, thus reducing the trust of developers during a software release as well as in the quality of a product. While rerunning test cases is a common approach, it is resource intensive, unreliable, and does not uncover the actual cause of test flakiness. Our paper evaluates an approach to identify randomness-related flaky. This paper used a divergence algorithm and execution tracing techniques to identify flaky tests, which resulted in the FlakyPy prototype. In addition, this paper discusses the cases where FlakyPy successfully identified the flaky test as well as those cases where FlakyPy failed. The papers discuss how the reporting mechanism of FlakyPy can help developers in identifying the root cause of randomness-related test flakiness. Thirty-two open-source projects were used in this. We concluded that FlakyPy can detect most of the randomness-related test flakiness. In addition, the reporting mechanism of FlakyPy reveals sufficient information about possible root causes of test flakiness.","PeriodicalId":147133,"journal":{"name":"2022 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)","volume":"52 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-04-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"131890704","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2022-04-01DOI: 10.1109/ICSTW55395.2022.00042
Thomas Laurent, Stephen Gaffney, Anthony Ventresque
Mutation analysis is a well-known testing criterion that involves seeding changes in the system under test, i.e. creating mutants, to simulate faults, and measuring the capacity of the test suite to detect these changes. The question of whether real faults are coupled with the mutants is central, as it determines whether tests that detect the mutants will also detect faults that actually occur in code, making the mutants reasonable test requirements. Prior work has explored this question, notably using the Defects4J dataset in Java. As the dataset and the mutation tools used in these prior works have evolved, this work re-visits this question using the newest available versions in order to strengthen and extend prior results. In this work we use 337 real faults from 15 different projects in the Defects4J 2.0.0 dataset, 2,828 test suites, and two well-known Java mutation testing tools (Major and Pitest) to explore (i) to what extent real faults are coupled with mutants, (ii) how both tools compare in terms of producing mutants coupled with faults, (iii) the characteristics of the mutants that are coupled with real faults, and (iv) the characteristics of faults not coupled with the mutants. Most (80.7%) of the faults used were coupled with at least one mutant created by Pitest or Major, most often with mutants created by both tools. All operators used produced a low (<4%) proportion of coupled mutants, although some operators are exclusively coupled to more faults, i.e. coupled to faults where no other operator produces coupled mutants. Finally, faults not coupled with any mutants usually had small fix patches, and although the code related to these faults was mostly affected by the mutation operators used the mutants produces were still not coupled. Results confirm previous findings showing that the coupling effect mostly holds but that additional mutation operators are needed to capture all faults.
{"title":"Re-visiting the coupling between mutants and real faults with Defects4J 2.0","authors":"Thomas Laurent, Stephen Gaffney, Anthony Ventresque","doi":"10.1109/ICSTW55395.2022.00042","DOIUrl":"https://doi.org/10.1109/ICSTW55395.2022.00042","url":null,"abstract":"Mutation analysis is a well-known testing criterion that involves seeding changes in the system under test, i.e. creating mutants, to simulate faults, and measuring the capacity of the test suite to detect these changes. The question of whether real faults are coupled with the mutants is central, as it determines whether tests that detect the mutants will also detect faults that actually occur in code, making the mutants reasonable test requirements. Prior work has explored this question, notably using the Defects4J dataset in Java. As the dataset and the mutation tools used in these prior works have evolved, this work re-visits this question using the newest available versions in order to strengthen and extend prior results. In this work we use 337 real faults from 15 different projects in the Defects4J 2.0.0 dataset, 2,828 test suites, and two well-known Java mutation testing tools (Major and Pitest) to explore (i) to what extent real faults are coupled with mutants, (ii) how both tools compare in terms of producing mutants coupled with faults, (iii) the characteristics of the mutants that are coupled with real faults, and (iv) the characteristics of faults not coupled with the mutants. Most (80.7%) of the faults used were coupled with at least one mutant created by Pitest or Major, most often with mutants created by both tools. All operators used produced a low (<4%) proportion of coupled mutants, although some operators are exclusively coupled to more faults, i.e. coupled to faults where no other operator produces coupled mutants. Finally, faults not coupled with any mutants usually had small fix patches, and although the code related to these faults was mostly affected by the mutation operators used the mutants produces were still not coupled. Results confirm previous findings showing that the coupling effect mostly holds but that additional mutation operators are needed to capture all faults.","PeriodicalId":147133,"journal":{"name":"2022 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)","volume":"71 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-04-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"133305643","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2022-04-01DOI: 10.1109/ICSTW55395.2022.00019
Wissam Mallouli
Software quality assurance (SQA) is a means and practice of monitoring the software engineering processes and methods used in a project to ensure proper quality of the software. It encompasses the entire software development life-cycle, including requirements engineering, software design, coding, source code reviews, software configuration management, testing , release management, software deployment and software integration. It is organized into goals, commitments, abilities, activities, measurements, verification and validation. In this talk, we will mainly focus on the testing activity part of the software development life-cycle. Its main objective is checking that software is satisfying a set of quality properties that are identified by the "ISO/IEC 25010:2011 System and Software Quality Model" standard [1] .
{"title":"Security Testing as part of Software Quality Assurance: Principles and Challenges","authors":"Wissam Mallouli","doi":"10.1109/ICSTW55395.2022.00019","DOIUrl":"https://doi.org/10.1109/ICSTW55395.2022.00019","url":null,"abstract":"Software quality assurance (SQA) is a means and practice of monitoring the software engineering processes and methods used in a project to ensure proper quality of the software. It encompasses the entire software development life-cycle, including requirements engineering, software design, coding, source code reviews, software configuration management, testing , release management, software deployment and software integration. It is organized into goals, commitments, abilities, activities, measurements, verification and validation. In this talk, we will mainly focus on the testing activity part of the software development life-cycle. Its main objective is checking that software is satisfying a set of quality properties that are identified by the \"ISO/IEC 25010:2011 System and Software Quality Model\" standard [1] .","PeriodicalId":147133,"journal":{"name":"2022 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)","volume":"71 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-04-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"114357653","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
京公网安备 11010802042870号
京ICP备2023020795号-1
扫码关注我们
求助内容：
应助结果提醒方式：