Pub Date : 2022-04-01DOI: 10.1109/ICSTW55395.2022.00029
Michael Wagner, Manuel Leithner, D. Simos, D. Kuhn, R. Kacker
CAgen is a state-of-the-art combinatorial test generation tool that is known for its execution speed. In addition, it supports an extensive list of features such as constraint handling, higher-index arrays, and import and export of models/test sets in various different formats. It is based on the FIPO algorithm, which can be considered an improved version of the widely used In-Parameter-Order strategy. In order to further speed up CAgen, this work first discusses how multithreading can be effectively used to optimally utilize available resources, particularly for large instances. We evaluate three different multithreaded variations of the horizontal extension and use the obtained insights to design the mFIPOG algorithm. In addition, we adopt methods that have previously been utilized to speed up constraint handling of CSP solvers in IPO algorithms into a forbidden tuple approach. In order to evaluate the performance of the improved tool, we provide results of benchmarks on the instances offered by the CT competition of IWCT 2022.
{"title":"Developing multithreaded techniques and improved constraint handling for the tool CAgen","authors":"Michael Wagner, Manuel Leithner, D. Simos, D. Kuhn, R. Kacker","doi":"10.1109/ICSTW55395.2022.00029","DOIUrl":"https://doi.org/10.1109/ICSTW55395.2022.00029","url":null,"abstract":"CAgen is a state-of-the-art combinatorial test generation tool that is known for its execution speed. In addition, it supports an extensive list of features such as constraint handling, higher-index arrays, and import and export of models/test sets in various different formats. It is based on the FIPO algorithm, which can be considered an improved version of the widely used In-Parameter-Order strategy. In order to further speed up CAgen, this work first discusses how multithreading can be effectively used to optimally utilize available resources, particularly for large instances. We evaluate three different multithreaded variations of the horizontal extension and use the obtained insights to design the mFIPOG algorithm. In addition, we adopt methods that have previously been utilized to speed up constraint handling of CSP solvers in IPO algorithms into a forbidden tuple approach. In order to evaluate the performance of the improved tool, we provide results of benchmarks on the instances offered by the CT competition of IWCT 2022.","PeriodicalId":147133,"journal":{"name":"2022 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)","volume":"15 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-04-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"128603168","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2022-04-01DOI: 10.1109/ICSTW55395.2022.00034
M. Abdelkarim, Reem ElAdawi
Regression testing is facing a bottleneck due to the growing number of test cases and the wide adoption of continuous integration (CI) in software projects, which increases the frequency of running software builds, making it challenging to run all the regression test cases. Machine learning (ML) techniques can be used to save time and hardware resources without compromising quality. In this work, we introduce a novel end-to-end, self-configurable, and incremental learning deep neural network (DNN) tool for test case prioritization (TCP-Net). TCP-Net is fed with source code-related features, test case metadata, test case coverage information, and test case failure history, to learn a high dimensional correlation between source files and test cases. We experimentally show that TCP-Net can be efficiently used for test case prioritization by evaluating it on three different real-life industrial software packages.
{"title":"TCP-Net: Test Case Prioritization using End-to-End Deep Neural Networks","authors":"M. Abdelkarim, Reem ElAdawi","doi":"10.1109/ICSTW55395.2022.00034","DOIUrl":"https://doi.org/10.1109/ICSTW55395.2022.00034","url":null,"abstract":"Regression testing is facing a bottleneck due to the growing number of test cases and the wide adoption of continuous integration (CI) in software projects, which increases the frequency of running software builds, making it challenging to run all the regression test cases. Machine learning (ML) techniques can be used to save time and hardware resources without compromising quality. In this work, we introduce a novel end-to-end, self-configurable, and incremental learning deep neural network (DNN) tool for test case prioritization (TCP-Net). TCP-Net is fed with source code-related features, test case metadata, test case coverage information, and test case failure history, to learn a high dimensional correlation between source files and test cases. We experimentally show that TCP-Net can be efficiently used for test case prioritization by evaluating it on three different real-life industrial software packages.","PeriodicalId":147133,"journal":{"name":"2022 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)","volume":"35 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-04-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"124984067","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2022-04-01DOI: 10.1109/ICSTW55395.2022.00050
Satoshi Masuda, J. Hagar, Y. Nishi, Kazuhiro Suzuki
A term, "software test architecture" has been discussed and defined. The discussion about analogy with software architecture is still remaining. Software architecture has been already defined and become a general term. In this paper, we discuss and define software test architecture as analogy with software architecture. Software architecture provides a well-developed framework for the structuring of specifications for large-scale, distributed systems. We define objective of software test architecture as helping for testing large-scale complex soft-ware by applying practices of software architecture to software testing.
{"title":"Software Test Architecture Definition by Analogy with Software Architecture","authors":"Satoshi Masuda, J. Hagar, Y. Nishi, Kazuhiro Suzuki","doi":"10.1109/ICSTW55395.2022.00050","DOIUrl":"https://doi.org/10.1109/ICSTW55395.2022.00050","url":null,"abstract":"A term, \"software test architecture\" has been discussed and defined. The discussion about analogy with software architecture is still remaining. Software architecture has been already defined and become a general term. In this paper, we discuss and define software test architecture as analogy with software architecture. Software architecture provides a well-developed framework for the structuring of specifications for large-scale, distributed systems. We define objective of software test architecture as helping for testing large-scale complex soft-ware by applying practices of software architecture to software testing.","PeriodicalId":147133,"journal":{"name":"2022 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)","volume":"51 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-04-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"126596896","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2022-04-01DOI: 10.1109/ICSTW55395.2022.00041
G. Guarnieri, A. V. Pizzoleto, F. Ferrari
This paper presents an implementation and assessment of a framework named SiMut. The framework was introduced in a previous paper with the objective of helping reducing the cost for testing a program based on groups of similar programs previously tested with mutation. The implementation presented in the paper handles Java programs and includes a set of variants that relate to three types of program abstraction (original source code, processed source code, and internal complexity metrics), three similarity calculation strategies (clustering, information diversity, and plagiarism), and one mutation cost reduction approach (inspired by the One-Op mutation technique). Our evaluation encompasses 20 variant combinations, also referred to as SiMut configurations, and 35 small Java programs. A cross-comparison involving the formed clusters and a comparison with randomly formed clusters points to configurations that tend to reach high effectiveness in foreseeing the best mutation operators for untested programs.
{"title":"An Automated Framework for Cost Reduction of Mutation Testing Based on Program Similarity","authors":"G. Guarnieri, A. V. Pizzoleto, F. Ferrari","doi":"10.1109/ICSTW55395.2022.00041","DOIUrl":"https://doi.org/10.1109/ICSTW55395.2022.00041","url":null,"abstract":"This paper presents an implementation and assessment of a framework named SiMut. The framework was introduced in a previous paper with the objective of helping reducing the cost for testing a program based on groups of similar programs previously tested with mutation. The implementation presented in the paper handles Java programs and includes a set of variants that relate to three types of program abstraction (original source code, processed source code, and internal complexity metrics), three similarity calculation strategies (clustering, information diversity, and plagiarism), and one mutation cost reduction approach (inspired by the One-Op mutation technique). Our evaluation encompasses 20 variant combinations, also referred to as SiMut configurations, and 35 small Java programs. A cross-comparison involving the formed clusters and a comparison with randomly formed clusters points to configurations that tend to reach high effectiveness in foreseeing the best mutation operators for untested programs.","PeriodicalId":147133,"journal":{"name":"2022 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)","volume":"18 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-04-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"123116577","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2022-04-01DOI: 10.1109/ICSTW55395.2022.00023
Giray Coşkun, Cankut Coşkun, Hanefi Mercan, Cemal Yilmaz
In this work, we express the masking and unique-cause MC/DC coverage criteria in Unified Combinatorial Interaction Testing (U-CIT) to test the interactions between the compile-time configuration options in highly configurable software systems. Our goal is not to propose yet another approach for achieving MC/DC coverage, but to further demonstrate the flexibility of U-CIT by applying it to a coverage criterion, which is quite different than the ones addressed by U-CIT so far. As the MC/DC criterion requires two test cases to show the independence of each condition, the test cases included in a U-CIT object cannot be generated independently from each other, which was the case for the coverage criteria addressed by U-CIT so far. Our empirical evaluations conducted on a dozen of highly configurable software systems demonstrate that U-CIT can flexibly address the aforementioned coverage criteria.
{"title":"Using Unified Combinatorial Interaction Testing for MC/DC Coverage","authors":"Giray Coşkun, Cankut Coşkun, Hanefi Mercan, Cemal Yilmaz","doi":"10.1109/ICSTW55395.2022.00023","DOIUrl":"https://doi.org/10.1109/ICSTW55395.2022.00023","url":null,"abstract":"In this work, we express the masking and unique-cause MC/DC coverage criteria in Unified Combinatorial Interaction Testing (U-CIT) to test the interactions between the compile-time configuration options in highly configurable software systems. Our goal is not to propose yet another approach for achieving MC/DC coverage, but to further demonstrate the flexibility of U-CIT by applying it to a coverage criterion, which is quite different than the ones addressed by U-CIT so far. As the MC/DC criterion requires two test cases to show the independence of each condition, the test cases included in a U-CIT object cannot be generated independently from each other, which was the case for the coverage criteria addressed by U-CIT so far. Our empirical evaluations conducted on a dozen of highly configurable software systems demonstrate that U-CIT can flexibly address the aforementioned coverage criteria.","PeriodicalId":147133,"journal":{"name":"2022 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)","volume":"52 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-04-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"129797550","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2022-04-01DOI: 10.1109/ICSTW55395.2022.00028
Yan Wang, Huayao Wu, Xintao Niu, Changhai Nie, Jiaxi Xu
Since Combinatorial Testing (CT) was first proposed in 1980s, there have been more than 50 algorithms or tools published for Covering Array Generation (CAG). This paper introduces the APPTS tool, which uses an Adaptive Penalty based Parallel Tabu Search (APPTS) algorithm to generate as small-sized Constrained Covering Arrays (CCA) as possible. Instead of simply reusing constraint solvers or forbidden tuple-based techniques to handle constraints, APPTS incorporates a penalty term into the fitness function to handle the constrained search space, and employs an adaptive penalty mechanism to dynamically adjust the penalty weight in different search phases. Our experiments on benchmarks of CT-competition demonstrate the effectiveness of the APPTS tool in covering array sizes. The APPTS tool executable is available at https://github.com/GIST-NJU/apptstool.
{"title":"A Constrained Covering Array Generator using Adaptive Penalty based Parallel Tabu Search","authors":"Yan Wang, Huayao Wu, Xintao Niu, Changhai Nie, Jiaxi Xu","doi":"10.1109/ICSTW55395.2022.00028","DOIUrl":"https://doi.org/10.1109/ICSTW55395.2022.00028","url":null,"abstract":"Since Combinatorial Testing (CT) was first proposed in 1980s, there have been more than 50 algorithms or tools published for Covering Array Generation (CAG). This paper introduces the APPTS tool, which uses an Adaptive Penalty based Parallel Tabu Search (APPTS) algorithm to generate as small-sized Constrained Covering Arrays (CCA) as possible. Instead of simply reusing constraint solvers or forbidden tuple-based techniques to handle constraints, APPTS incorporates a penalty term into the fitness function to handle the constrained search space, and employs an adaptive penalty mechanism to dynamically adjust the penalty weight in different search phases. Our experiments on benchmarks of CT-competition demonstrate the effectiveness of the APPTS tool in covering array sizes. The APPTS tool executable is available at https://github.com/GIST-NJU/apptstool.","PeriodicalId":147133,"journal":{"name":"2022 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)","volume":"45 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-04-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"134512168","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2022-04-01DOI: 10.1109/ICSTW55395.2022.00026
Kenya Takemura
The paper reports a C++ program for generating combinatorial test suites which I wrote for the purpose of learning combinatorial testing. The program is basically a direct implementation of the IPO strategy except for some optimization techniques. Unlike in the original strategy, for example, don’t cares are filled with some specific values that are heuristically chosen.
{"title":"A C++ implementation of the IPO algorithm","authors":"Kenya Takemura","doi":"10.1109/ICSTW55395.2022.00026","DOIUrl":"https://doi.org/10.1109/ICSTW55395.2022.00026","url":null,"abstract":"The paper reports a C++ program for generating combinatorial test suites which I wrote for the purpose of learning combinatorial testing. The program is basically a direct implementation of the IPO strategy except for some optimization techniques. Unlike in the original strategy, for example, don’t cares are filled with some specific values that are heuristically chosen.","PeriodicalId":147133,"journal":{"name":"2022 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)","volume":"35 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-04-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"130749748","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2022-04-01DOI: 10.1109/icstw55395.2022.00012
{"title":"Message from the Mutation 2022 Workshop Chairs","authors":"","doi":"10.1109/icstw55395.2022.00012","DOIUrl":"https://doi.org/10.1109/icstw55395.2022.00012","url":null,"abstract":"","PeriodicalId":147133,"journal":{"name":"2022 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)","volume":"1 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-04-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"130758073","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2022-04-01DOI: 10.1109/ICSTW55395.2022.00045
Tommaso Fulcini, Luca Ardito
In the context of Software Engineering, testing is a well-known phase that plays a critical role, as is needed to ensure that the designed and produced code provides the expected results, avoiding faults and crashes. Exploratory GUI testing allows the tester to manually define test cases by directly interacting with the user interface of the finite system. However, testers often loosely perform exploratory GUI testing, as they perceive it as a time-consuming, repetitive and unappealing activity. We defined a gamified framework for GUI testing to address this issue, which we developed and integrated into the Augmented testing tool, Scout. Gamification is perceived as a means to enhance the performance of human testers by stimulating competition and encouraging them to achieve better results in terms of both efficiency and effectiveness. We performed a preliminary evaluation of the gamification layer with a small sample of testers to assess the benefits of the technique compared with the standard version of the same tool. Test sequences defined with the gamified tool achieved higher coverage (i.e., higher efficiency) and a slightly higher percentage of bugs found. The user’s opinion was almost unanimously in favor of the gamified version of the tool.
{"title":"Gamified Exploratory GUI Testing of Web Applications: a Preliminary Evaluation","authors":"Tommaso Fulcini, Luca Ardito","doi":"10.1109/ICSTW55395.2022.00045","DOIUrl":"https://doi.org/10.1109/ICSTW55395.2022.00045","url":null,"abstract":"In the context of Software Engineering, testing is a well-known phase that plays a critical role, as is needed to ensure that the designed and produced code provides the expected results, avoiding faults and crashes. Exploratory GUI testing allows the tester to manually define test cases by directly interacting with the user interface of the finite system. However, testers often loosely perform exploratory GUI testing, as they perceive it as a time-consuming, repetitive and unappealing activity. We defined a gamified framework for GUI testing to address this issue, which we developed and integrated into the Augmented testing tool, Scout. Gamification is perceived as a means to enhance the performance of human testers by stimulating competition and encouraging them to achieve better results in terms of both efficiency and effectiveness. We performed a preliminary evaluation of the gamification layer with a small sample of testers to assess the benefits of the technique compared with the standard version of the same tool. Test sequences defined with the gamified tool achieved higher coverage (i.e., higher efficiency) and a slightly higher percentage of bugs found. The user’s opinion was almost unanimously in favor of the gamified version of the tool.","PeriodicalId":147133,"journal":{"name":"2022 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)","volume":"1 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-04-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"133275472","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Graphical user interface (GUI) significantly affects the lifetime of mobile applications. However, the diversity of mobile technologies, platforms, and devices makes the GUI compatibility testing of mobile applications a challenging task. In this paper, by leveraging computer vision techniques, we take the GUI compatibility testing display problem as a screenshot difference detection problem and propose an image-based method named Cross-Device Difference Detector (CdDiff) to help testers more efficiently and accurately improve mobile application compatibility. Specifically, CdDiff uses a pair of screenshots from two devices as inputs, employs a well-designed page understanding module to extract key screen elements from each screenshot, and then utilizes a discrepancy positioning module to detect and visualize the difference elements between the two screenshots in an easily recognizable way. In addition, we built a cross-device GUI compatibility testing dataset, which contains 725 pairs of user interface screenshots taken from Baidu Netdisk, Baidu App, and Baidu Tieba, displayed on ten popular smartphones. Experiments conducted on the built dataset show that the proposed CdDiff achieved an overall accuracy of 0.924, demonstrating its effectiveness.
{"title":"Cross-Device Difference Detector for Mobile Application GUI Compatibility Testing","authors":"Yanwei Ren, Youda Gu, Zongqing Ma, Hualiang Zhu, Fei Yin","doi":"10.1109/ICSTW55395.2022.00052","DOIUrl":"https://doi.org/10.1109/ICSTW55395.2022.00052","url":null,"abstract":"Graphical user interface (GUI) significantly affects the lifetime of mobile applications. However, the diversity of mobile technologies, platforms, and devices makes the GUI compatibility testing of mobile applications a challenging task. In this paper, by leveraging computer vision techniques, we take the GUI compatibility testing display problem as a screenshot difference detection problem and propose an image-based method named Cross-Device Difference Detector (CdDiff) to help testers more efficiently and accurately improve mobile application compatibility. Specifically, CdDiff uses a pair of screenshots from two devices as inputs, employs a well-designed page understanding module to extract key screen elements from each screenshot, and then utilizes a discrepancy positioning module to detect and visualize the difference elements between the two screenshots in an easily recognizable way. In addition, we built a cross-device GUI compatibility testing dataset, which contains 725 pairs of user interface screenshots taken from Baidu Netdisk, Baidu App, and Baidu Tieba, displayed on ten popular smartphones. Experiments conducted on the built dataset show that the proposed CdDiff achieved an overall accuracy of 0.924, demonstrating its effectiveness.","PeriodicalId":147133,"journal":{"name":"2022 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)","volume":"25 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-04-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"124370238","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
京公网安备 11010802042870号
京ICP备2023020795号-1
扫码关注我们
求助内容：
应助结果提醒方式：