Pub Date : 2021-05-01DOI: 10.1109/SBST52555.2021.00018
M. H. Moghadam, Markus Borg, S. J. Mousavirad
Deeper is a simulation-based test generator that uses an evolutionary process, i.e., an archive-based NSGA-II augmented with a quality population seed, for generating test cases to test a deep neural network-based lane-keeping system. This paper presents Deeper briefly and summarizes the results of Deeper's participation in the Cyber-physical systems (CPS) testing competition at SBST 2021.
{"title":"Deeper at the SBST 2021 Tool Competition: ADAS Testing Using Multi-Objective Search","authors":"M. H. Moghadam, Markus Borg, S. J. Mousavirad","doi":"10.1109/SBST52555.2021.00018","DOIUrl":"https://doi.org/10.1109/SBST52555.2021.00018","url":null,"abstract":"Deeper is a simulation-based test generator that uses an evolutionary process, i.e., an archive-based NSGA-II augmented with a quality population seed, for generating test cases to test a deep neural network-based lane-keeping system. This paper presents Deeper briefly and summarizes the results of Deeper's participation in the Cyber-physical systems (CPS) testing competition at SBST 2021.","PeriodicalId":199085,"journal":{"name":"2021 IEEE/ACM 14th International Workshop on Search-Based Software Testing (SBST)","volume":"33 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-05-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"125260172","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2021-05-01DOI: 10.1109/SBST52555.2021.00008
Byron DeVries, C. Trefftz
A common task in search-based testing is automatically identifying valuable test cases for software systems. However, existing approaches tend to either search for unique tests with regard to inputs or outputs (i.e., novelty search) or search for inputs that invalidate some expected proposition regarding the software (i.e., metamorphic testing). Problematically, verifying unique tests induces the oracle problem while an invalidated proposition results in a single test case. In this paper we utilize novelty search and metamorphic testing to discover a broad range of unique test cases that are directly verifiable via a metamorphic relation and invalidate such an expected proposition in fewer generations of an evolutionary algorithm than direct search. We apply this novelty search and metamorphic testing combination to discover errors in identifying the midpoint of a geodesic as a proof-of-concept.
{"title":"A Novelty Search and Metamorphic Testing Approach to Automatic Test Generation","authors":"Byron DeVries, C. Trefftz","doi":"10.1109/SBST52555.2021.00008","DOIUrl":"https://doi.org/10.1109/SBST52555.2021.00008","url":null,"abstract":"A common task in search-based testing is automatically identifying valuable test cases for software systems. However, existing approaches tend to either search for unique tests with regard to inputs or outputs (i.e., novelty search) or search for inputs that invalidate some expected proposition regarding the software (i.e., metamorphic testing). Problematically, verifying unique tests induces the oracle problem while an invalidated proposition results in a single test case. In this paper we utilize novelty search and metamorphic testing to discover a broad range of unique test cases that are directly verifiable via a metamorphic relation and invalidate such an expected proposition in fewer generations of an evolutionary algorithm than direct search. We apply this novelty search and metamorphic testing combination to discover errors in identifying the midpoint of a geodesic as a proof-of-concept.","PeriodicalId":199085,"journal":{"name":"2021 IEEE/ACM 14th International Workshop on Search-Based Software Testing (SBST)","volume":"4 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-05-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"126453135","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
We report on the organization, challenges, and results of the ninth edition of the Java Unit Testing Competition as well as the first edition of the Cyber-Physical Systems Testing Tool Competition. Java Unit Testing Competition. This year, five tools, Randoop, UtBot, Kex, Evosuite, and EvosuiteDSE, were executed on a benchmark with (i) new classes under test, selected from three open-source software projects, and (ii) the set of classes from three projects considered in the eighth edition. We relied on an improved Docker infrastructure to execute the tools and the subsequent coverage and mutation analysis. Given the high number of participants, we considered only two time budgets for test case generation: thirty seconds and two minutes. Cyber- Physical Systems Testing Tool Competition. Five tools, Deeper, Frenetic, GABExplore, GAB Exploit, and Swat, competed on testing self-driving car software by generating simulation-based tests using our new testing infrastructure. We considered two experimental settings to study test generators' transitory and asymptotic behaviors and evaluated the tools' test generation effectiveness and the exposed failures' diversity. This paper describes our methodology, the statistical analysis of the results together with the contestant tools, and the challenges faced while running the competition experiments.
{"title":"SBST Tool Competition 2021","authors":"Sebastiano Panichella, Alessio Gambi, Fiorella Zampetti, Vincenzo Riccio","doi":"10.1109/SBST52555.2021.00011","DOIUrl":"https://doi.org/10.1109/SBST52555.2021.00011","url":null,"abstract":"We report on the organization, challenges, and results of the ninth edition of the Java Unit Testing Competition as well as the first edition of the Cyber-Physical Systems Testing Tool Competition. Java Unit Testing Competition. This year, five tools, Randoop, UtBot, Kex, Evosuite, and EvosuiteDSE, were executed on a benchmark with (i) new classes under test, selected from three open-source software projects, and (ii) the set of classes from three projects considered in the eighth edition. We relied on an improved Docker infrastructure to execute the tools and the subsequent coverage and mutation analysis. Given the high number of participants, we considered only two time budgets for test case generation: thirty seconds and two minutes. Cyber- Physical Systems Testing Tool Competition. Five tools, Deeper, Frenetic, GABExplore, GAB Exploit, and Swat, competed on testing self-driving car software by generating simulation-based tests using our new testing infrastructure. We considered two experimental settings to study test generators' transitory and asymptotic behaviors and evaluated the tools' test generation effectiveness and the exposed failures' diversity. This paper describes our methodology, the statistical analysis of the results together with the contestant tools, and the challenges faced while running the competition experiments.","PeriodicalId":199085,"journal":{"name":"2021 IEEE/ACM 14th International Workshop on Search-Based Software Testing (SBST)","volume":"10 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-05-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"131269371","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2021-05-01DOI: 10.1109/SBST52555.2021.00013
Ignacio Manuel Lebrero Rial, Juan P. Galeotti
EvoSuite is a search-based tool with a dynamic symbolic execution module (EvosuiteDSE) that automatically generates executable unit tests for java code (JUnit tests). This paper summarizes the results and experiences of EvoSuiteDSE's participation at the ninth unit testing competition at SBST 2021, where EvoSuiteDSE achieve an overall score of 47.14 in its first participation on the competition.
{"title":"EvoSuiteDSE at the SBST 2021 Tool Competition","authors":"Ignacio Manuel Lebrero Rial, Juan P. Galeotti","doi":"10.1109/SBST52555.2021.00013","DOIUrl":"https://doi.org/10.1109/SBST52555.2021.00013","url":null,"abstract":"EvoSuite is a search-based tool with a dynamic symbolic execution module (EvosuiteDSE) that automatically generates executable unit tests for java code (JUnit tests). This paper summarizes the results and experiences of EvoSuiteDSE's participation at the ninth unit testing competition at SBST 2021, where EvoSuiteDSE achieve an overall score of 47.14 in its first participation on the competition.","PeriodicalId":199085,"journal":{"name":"2021 IEEE/ACM 14th International Workshop on Search-Based Software Testing (SBST)","volume":"30 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-05-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"134243865","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2021-05-01DOI: 10.1109/SBST52555.2021.00017
Florian Klück, Lorenz Klampfl, F. Wotawa
GABezier is a search-based tool for the automatic generation of challenging road networks for virtual testing of an automated lane keep system (ALKS). This paper provides a brief overview on the tool and summarizes the results of GABezier's participation at the first edition of the Cyber-Physical Systems Testing Tool Competition. We submitted our tool in two configurations, namely GABExplore and GABExploit. Especially the latter configuration has efficiently generated valid test cases and triggered many faults.
{"title":"GABezier at the SBST 2021 Tool Competition","authors":"Florian Klück, Lorenz Klampfl, F. Wotawa","doi":"10.1109/SBST52555.2021.00017","DOIUrl":"https://doi.org/10.1109/SBST52555.2021.00017","url":null,"abstract":"GABezier is a search-based tool for the automatic generation of challenging road networks for virtual testing of an automated lane keep system (ALKS). This paper provides a brief overview on the tool and summarizes the results of GABezier's participation at the first edition of the Cyber-Physical Systems Testing Tool Competition. We submitted our tool in two configurations, namely GABExplore and GABExploit. Especially the latter configuration has efficiently generated valid test cases and triggered many faults.","PeriodicalId":199085,"journal":{"name":"2021 IEEE/ACM 14th International Workshop on Search-Based Software Testing (SBST)","volume":"33 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-05-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"130561793","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2021-05-01DOI: 10.1109/SBST52555.2021.00012
S. Vogl, Sebastian Schweikl, G. Fraser, Andrea Arcuri, José Campos, Annibale Panichella
EvoSuite is a search-based unit test generation tool for Java. This paper summarises the results and experiences of EvoSuite's participation at the ninth unit testing competition at SBST 2021, where EvoSuite achieved the highest overall score.
{"title":"Evosuite at the SBST 2021 Tool Competition","authors":"S. Vogl, Sebastian Schweikl, G. Fraser, Andrea Arcuri, José Campos, Annibale Panichella","doi":"10.1109/SBST52555.2021.00012","DOIUrl":"https://doi.org/10.1109/SBST52555.2021.00012","url":null,"abstract":"EvoSuite is a search-based unit test generation tool for Java. This paper summarises the results and experiences of EvoSuite's participation at the ninth unit testing competition at SBST 2021, where EvoSuite achieved the highest overall score.","PeriodicalId":199085,"journal":{"name":"2021 IEEE/ACM 14th International Workshop on Search-Based Software Testing (SBST)","volume":"138 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-05-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"114170933","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2021-05-01DOI: 10.1109/SBST52555.2021.00016
Ezequiel Castellano, A. Cetinkaya, Cédric Ho Thanh, Stefan Klikovits, Xiaoyi Zhang, Paolo Arcaini
Frenetic is a genetic approach that leverages a curvature-based road representation. Given an autonomous driving agent, the goal of Frenetic is to generate roads where the agent fails to stay within its lane. In other words, Frenetic tries to minimize the “out of bound distance”, which is the distance between the car and either edge of the lane if the car is within the lane, and proceeds to negative values once the car drives off. This work resembles classic aspects of genetic algorithms such as mutations and crossover, but introduces some nuances aiming at improving diversity of the generated roads.
{"title":"Frenetic at the SBST 2021 Tool Competition","authors":"Ezequiel Castellano, A. Cetinkaya, Cédric Ho Thanh, Stefan Klikovits, Xiaoyi Zhang, Paolo Arcaini","doi":"10.1109/SBST52555.2021.00016","DOIUrl":"https://doi.org/10.1109/SBST52555.2021.00016","url":null,"abstract":"Frenetic is a genetic approach that leverages a curvature-based road representation. Given an autonomous driving agent, the goal of Frenetic is to generate roads where the agent fails to stay within its lane. In other words, Frenetic tries to minimize the “out of bound distance”, which is the distance between the car and either edge of the lane if the car is within the lane, and proceeds to negative values once the car drives off. This work resembles classic aspects of genetic algorithms such as mutations and crossover, but introduces some nuances aiming at improving diversity of the generated roads.","PeriodicalId":199085,"journal":{"name":"2021 IEEE/ACM 14th International Workshop on Search-Based Software Testing (SBST)","volume":"9 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-05-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"121647980","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2021-05-01DOI: 10.1109/SBST52555.2021.00009
Paulo Canelas, José Campos, C. Timperley, Alcides Fonseca
Automated test generation helps programmers to test their software with minimal intervention. Automated test generation tools produce a set of program inputs that maximize the possible execution paths, presented as a test coverage metric. Proposed approaches fall within three main approaches. Search-based methods work on any program by randomly searching for inputs that maximize coverage. Heuristic-based methods can be used to have better performance than pure random-search. Constraint-based methods use symbolic execution to restrict the random inputs to those guaranteed to explore different paths. Despite making the execution slower and supporting very few programs, these methods are more efficient because the search space is vastly reduced. The third approach combines the previous two to support any program and takes advantage of the space search reduction when able, at the cost of slower execution. We propose a fourth approach that also refines search-based with constraints. However, instead of requiring a slower symbolic execution when measuring coverage, constraints are statically extracted from the source code before the search procedure occurs. Our approach supports all programs (as in Search-Based) and reduces the search-space (as in Constraint-based methods). The innovation is that static analysis occurs only once and, despite being less exact that symbolic execution, it can significantly reduce the execution cost in every coverage measurement. This paper introduces this approach, describes how it can be implemented and discusses its advantages and drawbacks.
{"title":"Augmenting Search-based Techniques with Static Synthesis-based Input Generation","authors":"Paulo Canelas, José Campos, C. Timperley, Alcides Fonseca","doi":"10.1109/SBST52555.2021.00009","DOIUrl":"https://doi.org/10.1109/SBST52555.2021.00009","url":null,"abstract":"Automated test generation helps programmers to test their software with minimal intervention. Automated test generation tools produce a set of program inputs that maximize the possible execution paths, presented as a test coverage metric. Proposed approaches fall within three main approaches. Search-based methods work on any program by randomly searching for inputs that maximize coverage. Heuristic-based methods can be used to have better performance than pure random-search. Constraint-based methods use symbolic execution to restrict the random inputs to those guaranteed to explore different paths. Despite making the execution slower and supporting very few programs, these methods are more efficient because the search space is vastly reduced. The third approach combines the previous two to support any program and takes advantage of the space search reduction when able, at the cost of slower execution. We propose a fourth approach that also refines search-based with constraints. However, instead of requiring a slower symbolic execution when measuring coverage, constraints are statically extracted from the source code before the search procedure occurs. Our approach supports all programs (as in Search-Based) and reduces the search-space (as in Constraint-based methods). The innovation is that static analysis occurs only once and, despite being less exact that symbolic execution, it can significantly reduce the execution cost in every coverage measurement. This paper introduces this approach, describes how it can be implemented and discusses its advantages and drawbacks.","PeriodicalId":199085,"journal":{"name":"2021 IEEE/ACM 14th International Workshop on Search-Based Software Testing (SBST)","volume":"42 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-05-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"114536082","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
UtBot is an automatic test generator for Java programs developed by Huawei and based on symbolic execution. It tries to cover as many branches as possible using the program's bytecode. To do that UtBot analyzes paths in the control flow graph of a given method, constructing constraints for them, and tries to find satisfying input values using SMT-solver to cover corresponding branches. In this paper, we report the results of UtBot at the ninth edition of the SBST 2021 tool competition.
{"title":"UtBot at the SBST2021 Tool Competition","authors":"Dmitry Ivanov, Nikolay Bukharev, Alexey Menshutin, Arsen Nagdalian, Gleb Stromov, Artem Ustinov","doi":"10.1109/SBST52555.2021.00015","DOIUrl":"https://doi.org/10.1109/SBST52555.2021.00015","url":null,"abstract":"UtBot is an automatic test generator for Java programs developed by Huawei and based on symbolic execution. It tries to cover as many branches as possible using the program's bytecode. To do that UtBot analyzes paths in the control flow graph of a given method, constructing constraints for them, and tries to find satisfying input values using SMT-solver to cover corresponding branches. In this paper, we report the results of UtBot at the ninth edition of the SBST 2021 tool competition.","PeriodicalId":199085,"journal":{"name":"2021 IEEE/ACM 14th International Workshop on Search-Based Software Testing (SBST)","volume":"13 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-05-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"130883745","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2021-05-01DOI: 10.1109/SBST52555.2021.00007
Alexandre Bergel, Ignacio Slater Muñoz
Software crashes are a problem all developers face eventually. Manually reproducing crashes can be very expensive and require a lot of effort. Recent studies have proposed techniques to automatically generate tests to detect and reproduce errors. But even if this topic has been widely studied, there has been little to no progress done for dynamically typed languages. This becomes important because current approaches take advantage of the type information inherent to statically typed languages to generate the sequence of instructions needed to reproduce a crash, thus making it unclear to judge if type information is necessary to reproduce errors. The lack of explicit type declarations in dynamic languages difficults the task of generating the instructions to replicate an error because the type checking can only be done during runtime, making algorithms less knowledgeable about the program and, therefore, making it more difficult to use search-based approaches because the algorithms have less information to work with. This paper presents a Genetic Algorithm approach to produce crash reproductions on Python based only on the information contained in the error's stack-trace. An empirical study analysing three different experiments were evaluated giving mostly positive results, achieving a high precision while reproducing the desired crashes (over 70%). The study shows that the presented approach is independent of the kind of typing of the language, and provides a solid base to further develop the topic.
{"title":"Beacon: Automated Test Generation for Stack-Trace Reproduction using Genetic Algorithms","authors":"Alexandre Bergel, Ignacio Slater Muñoz","doi":"10.1109/SBST52555.2021.00007","DOIUrl":"https://doi.org/10.1109/SBST52555.2021.00007","url":null,"abstract":"Software crashes are a problem all developers face eventually. Manually reproducing crashes can be very expensive and require a lot of effort. Recent studies have proposed techniques to automatically generate tests to detect and reproduce errors. But even if this topic has been widely studied, there has been little to no progress done for dynamically typed languages. This becomes important because current approaches take advantage of the type information inherent to statically typed languages to generate the sequence of instructions needed to reproduce a crash, thus making it unclear to judge if type information is necessary to reproduce errors. The lack of explicit type declarations in dynamic languages difficults the task of generating the instructions to replicate an error because the type checking can only be done during runtime, making algorithms less knowledgeable about the program and, therefore, making it more difficult to use search-based approaches because the algorithms have less information to work with. This paper presents a Genetic Algorithm approach to produce crash reproductions on Python based only on the information contained in the error's stack-trace. An empirical study analysing three different experiments were evaluated giving mostly positive results, achieving a high precision while reproducing the desired crashes (over 70%). The study shows that the presented approach is independent of the kind of typing of the language, and provides a solid base to further develop the topic.","PeriodicalId":199085,"journal":{"name":"2021 IEEE/ACM 14th International Workshop on Search-Based Software Testing (SBST)","volume":"545 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-05-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"116510575","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
京公网安备 11010802042870号
京ICP备2023020795号-1
扫码关注我们
求助内容：
应助结果提醒方式：