Pub Date : 2022-04-01DOI: 10.1109/ICSTW55395.2022.00037
Cristina Gatt, Mark Bugeja, Mark Micallef
When setting out a research roadmap for software testing, Bertolino [1] presented four dreams, one of which was 100% automatic testing. Fifteen years later, the dream has not been realised but the promise of artificial intelligence techniques brings us closer than ever before. In this paper, we propose that one way to achieve this goal is to leverage the commonalities that exist amongst domain-specific applications. That is to say that whilst every application within a particular domain is arguably unique, they all share a considerable overlap in terms of features.We propose an approach based on Behavioural Cloning, an AI technique whereby an agent observes traces by an expert and attempts to carry out domain-specific tasks in previously unseen contexts based on those traces. Using online stores as a case study, we discuss initial investigations into this idea, present results and identify a roadmap going forward.
{"title":"Towards Domain-Specific Automated Testing via Behavioural Cloning","authors":"Cristina Gatt, Mark Bugeja, Mark Micallef","doi":"10.1109/ICSTW55395.2022.00037","DOIUrl":"https://doi.org/10.1109/ICSTW55395.2022.00037","url":null,"abstract":"When setting out a research roadmap for software testing, Bertolino [1] presented four dreams, one of which was 100% automatic testing. Fifteen years later, the dream has not been realised but the promise of artificial intelligence techniques brings us closer than ever before. In this paper, we propose that one way to achieve this goal is to leverage the commonalities that exist amongst domain-specific applications. That is to say that whilst every application within a particular domain is arguably unique, they all share a considerable overlap in terms of features.We propose an approach based on Behavioural Cloning, an AI technique whereby an agent observes traces by an expert and attempts to carry out domain-specific tasks in previously unseen contexts based on those traces. Using online stores as a case study, we discuss initial investigations into this idea, present results and identify a roadmap going forward.","PeriodicalId":147133,"journal":{"name":"2022 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)","volume":"46 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-04-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"116421750","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2022-04-01DOI: 10.1109/ICSTW55395.2022.00040
Rowland Pitts
Mutation Testing offers a powerful approach to assessing unit test set quality; however, software developers are often reluctant to embrace the technique because of the tremendous number of mutants it generates, including redundant and equivalent mutants. In spite of their large numbers, redundant mutants are relatively innocuous. Recent research indicates that redundant mutants affect a test engineer’s work effort only slightly, whereas equivalent mutants have a direct linear impact. Moreover, the time invested analyzing equivalent mutants produces no unit tests. Dominator mutants address the redundancy problem by allowing the tester to write a minimal number of tests. This paper demonstrates empirically that when using randomly selected mutants as test requirements, the probability of drawing a dominator or quasi-dominator is initially double that of a non-dominator, and progressively increases. It also demonstrates that even non-dominator mutants are highly likely to elicit dominator killing tests. Finally it demonstrates that the probability of selecting an equivalent mutant quickly overwhelms all other selections. These observations provide new insight into the effectiveness of random mutant selection, and into the magnitude of the problem posed by equivalent mutants.
{"title":"Random Mutant Selection and Equivalent Mutants Revisited","authors":"Rowland Pitts","doi":"10.1109/ICSTW55395.2022.00040","DOIUrl":"https://doi.org/10.1109/ICSTW55395.2022.00040","url":null,"abstract":"Mutation Testing offers a powerful approach to assessing unit test set quality; however, software developers are often reluctant to embrace the technique because of the tremendous number of mutants it generates, including redundant and equivalent mutants. In spite of their large numbers, redundant mutants are relatively innocuous. Recent research indicates that redundant mutants affect a test engineer’s work effort only slightly, whereas equivalent mutants have a direct linear impact. Moreover, the time invested analyzing equivalent mutants produces no unit tests. Dominator mutants address the redundancy problem by allowing the tester to write a minimal number of tests. This paper demonstrates empirically that when using randomly selected mutants as test requirements, the probability of drawing a dominator or quasi-dominator is initially double that of a non-dominator, and progressively increases. It also demonstrates that even non-dominator mutants are highly likely to elicit dominator killing tests. Finally it demonstrates that the probability of selecting an equivalent mutant quickly overwhelms all other selections. These observations provide new insight into the effectiveness of random mutant selection, and into the magnitude of the problem posed by equivalent mutants.","PeriodicalId":147133,"journal":{"name":"2022 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)","volume":"1 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-04-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"129098943","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2022-04-01DOI: 10.1109/ICSTW55395.2022.00033
Miguel Trinca, J. Ferreira, Rui Abreu
Generative Sequence-To-Sequence models have been proposed for the task of generating well-formed programs, an important task for fuzz testing tools such as compilers. In this paper, we propose a Sequence-to-Sequence model to generate well-formed Q# Quantum programs. The ratio of syntactically valid programs among 1,000 Q# files generated by our model is 79.6%. In addition, we also contribute with a dataset of 1,723 Q# files taken from publicly available repositories on GitHub, which can be used by the growing community of Quantum Software Engineering.
{"title":"A Preliminary Study on Generating Well-Formed Q# Quantum Programs for Fuzz Testing","authors":"Miguel Trinca, J. Ferreira, Rui Abreu","doi":"10.1109/ICSTW55395.2022.00033","DOIUrl":"https://doi.org/10.1109/ICSTW55395.2022.00033","url":null,"abstract":"Generative Sequence-To-Sequence models have been proposed for the task of generating well-formed programs, an important task for fuzz testing tools such as compilers. In this paper, we propose a Sequence-to-Sequence model to generate well-formed Q# Quantum programs. The ratio of syntactically valid programs among 1,000 Q# files generated by our model is 79.6%. In addition, we also contribute with a dataset of 1,723 Q# files taken from publicly available repositories on GitHub, which can be used by the growing community of Quantum Software Engineering.","PeriodicalId":147133,"journal":{"name":"2022 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)","volume":"64 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-04-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"114215847","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2022-04-01DOI: 10.1109/ICSTW55395.2022.00027
A. Bombarda, A. Gargantini
Combinatorial interaction testing (CIT) is a testing technique that has proved to be effective in finding faults due to the interaction among inputs, and in reducing the number of test cases. One of the most crucial parts of combinatorial testing is the test generation for which many tools and algorithms have been proposed in recent years, with different methodologies and performances. However, generating tests remains a complex procedure that can require a lot of effort (mainly time). Thus, in this paper, we present the tool pMEDICI which aims to reduce the test generation time by parallelizing the generation process and exploiting the recent multithread hardware architectures. It uses Multivalued Decision Diagrams (MDDs) for representing the constraints and the tuples to be tested and extracts from them the t-wise test cases. Our experiments confirm that our tool requires a shorter amount of time for generating combinatorial test suites, especially for complex models, with a lot of parameters and constraints.
{"title":"Parallel Test Generation for Combinatorial Models Based on Multivalued Decision Diagrams","authors":"A. Bombarda, A. Gargantini","doi":"10.1109/ICSTW55395.2022.00027","DOIUrl":"https://doi.org/10.1109/ICSTW55395.2022.00027","url":null,"abstract":"Combinatorial interaction testing (CIT) is a testing technique that has proved to be effective in finding faults due to the interaction among inputs, and in reducing the number of test cases. One of the most crucial parts of combinatorial testing is the test generation for which many tools and algorithms have been proposed in recent years, with different methodologies and performances. However, generating tests remains a complex procedure that can require a lot of effort (mainly time). Thus, in this paper, we present the tool pMEDICI which aims to reduce the test generation time by parallelizing the generation process and exploiting the recent multithread hardware architectures. It uses Multivalued Decision Diagrams (MDDs) for representing the constraints and the tuples to be tested and extracts from them the t-wise test cases. Our experiments confirm that our tool requires a shorter amount of time for generating combinatorial test suites, especially for complex models, with a lot of parameters and constraints.","PeriodicalId":147133,"journal":{"name":"2022 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)","volume":"47 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-04-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"133639710","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2022-04-01DOI: 10.1109/ICSTW55395.2022.00016
Manh-Dung Nguyen, Vinh Hoa La, A. Cavalli, Edgardo Montes de Oca
Artificial Intelligence (AI) is envisioned to play a critical role in controlling and orchestrating 5G/IoT networks and their applications, thanks to its capabilities to recognize abnormal patterns in complex situations and produce accurate decisions. However, AI models are vulnerable to adversarial attacks, thus the societal view is far from trustworthy as to its usage in safety critical areas relying on 5G/IoT networks. In this paper, we present ongoing work being done in the H2020 SPATIAL project that targets developing and evaluating AI-based modules for anomaly detection and Root Cause Analysis in the 5G/IoT context regarding different criteria, such as explainability, resilience and performance on a real 5G/IoT testbed.
{"title":"Towards improving explainability, resilience and performance of cybersecurity analysis of 5G/IoT networks (work-in-progress paper)","authors":"Manh-Dung Nguyen, Vinh Hoa La, A. Cavalli, Edgardo Montes de Oca","doi":"10.1109/ICSTW55395.2022.00016","DOIUrl":"https://doi.org/10.1109/ICSTW55395.2022.00016","url":null,"abstract":"Artificial Intelligence (AI) is envisioned to play a critical role in controlling and orchestrating 5G/IoT networks and their applications, thanks to its capabilities to recognize abnormal patterns in complex situations and produce accurate decisions. However, AI models are vulnerable to adversarial attacks, thus the societal view is far from trustworthy as to its usage in safety critical areas relying on 5G/IoT networks. In this paper, we present ongoing work being done in the H2020 SPATIAL project that targets developing and evaluating AI-based modules for anomaly detection and Root Cause Analysis in the 5G/IoT context regarding different criteria, such as explainability, resilience and performance on a real 5G/IoT testbed.","PeriodicalId":147133,"journal":{"name":"2022 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)","volume":"17 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-04-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"123821448","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2022-04-01DOI: 10.1109/icstw55395.2022.00011
{"title":"Message from the InSTA 2022 chairs","authors":"","doi":"10.1109/icstw55395.2022.00011","DOIUrl":"https://doi.org/10.1109/icstw55395.2022.00011","url":null,"abstract":"","PeriodicalId":147133,"journal":{"name":"2022 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)","volume":"26 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-04-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"125951449","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2022-04-01DOI: 10.1109/ICSTW55395.2022.00054
You Wu, Qi Zhan, Haipeng Qu, Xiaoqi Zhao
In recent years, coverage-based greybox fuzzing (CGF) has become one of the most important techniques to discover security bugs. The existing fuzzers score the seeds, and then allocate the energy to the seeds according to the scoring results, but most seeds obtain the same energy, and then repeatedly select the same seeds for fuzzing. These strategies have been proved to be inefficient. Our experimental observations show that various seeds have diverse efficiency, and the efficiency of test cases changes increase with execution time. In this paper, we propose a novel yet lightweight energy allocation strategy, called AcoFuzz, to improve fuzzing efficiency. AcoFuzz has one following distinct advantage: Dynamically allocate energy for seeds to cope with their efficiency variation. Extensive experiments based on real-world programs and the LAVA-M dataset have been conducted to evaluate the path discovery and vulnerability detection ability of AcoFuzz, which substantially outperforms 3 state-of-the-art fuzzers.
{"title":"AcoFuzz: Adaptive Energy Allocation for Greybox Fuzzing","authors":"You Wu, Qi Zhan, Haipeng Qu, Xiaoqi Zhao","doi":"10.1109/ICSTW55395.2022.00054","DOIUrl":"https://doi.org/10.1109/ICSTW55395.2022.00054","url":null,"abstract":"In recent years, coverage-based greybox fuzzing (CGF) has become one of the most important techniques to discover security bugs. The existing fuzzers score the seeds, and then allocate the energy to the seeds according to the scoring results, but most seeds obtain the same energy, and then repeatedly select the same seeds for fuzzing. These strategies have been proved to be inefficient. Our experimental observations show that various seeds have diverse efficiency, and the efficiency of test cases changes increase with execution time. In this paper, we propose a novel yet lightweight energy allocation strategy, called AcoFuzz, to improve fuzzing efficiency. AcoFuzz has one following distinct advantage: Dynamically allocate energy for seeds to cope with their efficiency variation. Extensive experiments based on real-world programs and the LAVA-M dataset have been conducted to evaluate the path discovery and vulnerability detection ability of AcoFuzz, which substantially outperforms 3 state-of-the-art fuzzers.","PeriodicalId":147133,"journal":{"name":"2022 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)","volume":"92 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-04-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"124601545","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2022-04-01DOI: 10.1109/ICSTW55395.2022.00043
M. Holmberg, Felix Dobslaw
Automating regression tests can bring many benefits, such as increased testing frequency and improved bug-finding capabilities, leading to an overall positive impact on software quality. However, transitioning manual testing into automated testing is not possible without effort. In this work, we attempted to transition a manual test suite into an automated graphical user interface-based regression test suite using a Robotic Process Automation framework. The study reports on the efforts needed to implement the test suite and test case maintenance efforts. Furthermore, the study reports on bug-finding capabilities and discusses the feasibility of applying Robotic Process Automation for automated graphical user interface-based regression testing more broadly. Due to challenges related to current testing practices within the organization, only a small subset of the manual test cases could be successfully transitioned. The results indicate that the implementation and maintenance efforts patterns are similar to those of previous studies from the literature - even similar benefits and challenges could be observed. These findings suggest that Robotic Process Automation is feasible for graphical user interface-based regression testing, but further (long-term) investigations are needed in this area.
{"title":"An Industrial Case-Study on GUI Testing With RPA","authors":"M. Holmberg, Felix Dobslaw","doi":"10.1109/ICSTW55395.2022.00043","DOIUrl":"https://doi.org/10.1109/ICSTW55395.2022.00043","url":null,"abstract":"Automating regression tests can bring many benefits, such as increased testing frequency and improved bug-finding capabilities, leading to an overall positive impact on software quality. However, transitioning manual testing into automated testing is not possible without effort. In this work, we attempted to transition a manual test suite into an automated graphical user interface-based regression test suite using a Robotic Process Automation framework. The study reports on the efforts needed to implement the test suite and test case maintenance efforts. Furthermore, the study reports on bug-finding capabilities and discusses the feasibility of applying Robotic Process Automation for automated graphical user interface-based regression testing more broadly. Due to challenges related to current testing practices within the organization, only a small subset of the manual test cases could be successfully transitioned. The results indicate that the implementation and maintenance efforts patterns are similar to those of previous studies from the literature - even similar benefits and challenges could be observed. These findings suggest that Robotic Process Automation is feasible for graphical user interface-based regression testing, but further (long-term) investigations are needed in this area.","PeriodicalId":147133,"journal":{"name":"2022 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)","volume":"150 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-04-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"128660507","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2022-04-01DOI: 10.1109/ICSTW55395.2022.00024
Manuel Leithner, D. Simos
Covering arrays (CAs) are combinatorial structures that play a significant role in software testing. While a sizable body of research has been dedicated to the generation of CAs, virtually no attention has been directed to their efficient storage, archival, and retrieval. Considering the potential economic advantages offered by the use of these structures, this is surprising: While the algorithmic complexity of most known methods used in their construction tends to be high, leading to expensive computations, the cost of storage has decreased in recent years. It thus seems prudent to identify and tackle the practical issues of large-scale and long-term archival and compression of such objects. This paper introduces CA2, the Compressed Covering Array Archive format, which offers competitive compression combined with CA-specific metadata that allows for targeted retrieval based on input specifications given in one of a variety of popular formats. An open source prototype implementation is provided as a utility for practitioners and building block for future research.
{"title":"CA2: Practical Archival and Compression of Covering Arrays","authors":"Manuel Leithner, D. Simos","doi":"10.1109/ICSTW55395.2022.00024","DOIUrl":"https://doi.org/10.1109/ICSTW55395.2022.00024","url":null,"abstract":"Covering arrays (CAs) are combinatorial structures that play a significant role in software testing. While a sizable body of research has been dedicated to the generation of CAs, virtually no attention has been directed to their efficient storage, archival, and retrieval. Considering the potential economic advantages offered by the use of these structures, this is surprising: While the algorithmic complexity of most known methods used in their construction tends to be high, leading to expensive computations, the cost of storage has decreased in recent years. It thus seems prudent to identify and tackle the practical issues of large-scale and long-term archival and compression of such objects. This paper introduces CA2, the Compressed Covering Array Archive format, which offers competitive compression combined with CA-specific metadata that allows for targeted retrieval based on input specifications given in one of a variety of popular formats. An open source prototype implementation is provided as a utility for practitioners and building block for future research.","PeriodicalId":147133,"journal":{"name":"2022 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)","volume":"26 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-04-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"126089844","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2022-04-01DOI: 10.1109/icstw55395.2022.00008
{"title":"General Message from the IWCT 2022 Workshop Chairs","authors":"","doi":"10.1109/icstw55395.2022.00008","DOIUrl":"https://doi.org/10.1109/icstw55395.2022.00008","url":null,"abstract":"","PeriodicalId":147133,"journal":{"name":"2022 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)","volume":"32 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-04-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"127946060","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
京公网安备 11010802042870号
京ICP备2023020795号-1
扫码关注我们
求助内容：
应助结果提醒方式：