Multi-agent systems (MAS) may encounter uncertainties in the form of unexpected environmental conditions, sub-optimal system configurations, and unplanned interactions between autonomous agents. The number of combinations of such uncertainties may be innumerable, however run-time testing may reduce the issues impacting such a system. We posit that search heuristics can augment a run-time testing process, in-situ, for a MAS. To support our position we discuss our in-progress experimental testbed to realize this goal and highlight challenges we anticipate for this domain.
{"title":"Towards Run-Time Search for Real-World Multi-Agent Systems","authors":"Abigail C. Diller, Erik M. Fredericks","doi":"10.1145/3526072.3527537","DOIUrl":"https://doi.org/10.1145/3526072.3527537","url":null,"abstract":"Multi-agent systems (MAS) may encounter uncertainties in the form of unexpected environmental conditions, sub-optimal system configurations, and unplanned interactions between autonomous agents. The number of combinations of such uncertainties may be innumerable, however run-time testing may reduce the issues impacting such a system. We posit that search heuristics can augment a run-time testing process, in-situ, for a MAS. To support our position we discuss our in-progress experimental testbed to realize this goal and highlight challenges we anticipate for this domain.","PeriodicalId":206275,"journal":{"name":"2022 IEEE/ACM 15th International Workshop on Search-Based Software Testing (SBST)","volume":"13 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-05-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"115242755","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
GenRL is a Deep Reinforcement Learning-based tool designed to generate test cases for Lane-Keeping Assist Systems. In this paper, we briefly presents GenRL, and summarize the results of its participation in the Cyber-Physical Systems (CPS) tool competition at SBST 2022.
{"title":"GenRL at the SBST 2022 Tool Competition","authors":"L. L. L. Starace, Andrea Romdhana, S. Martino","doi":"10.1145/3526072.3527533","DOIUrl":"https://doi.org/10.1145/3526072.3527533","url":null,"abstract":"GenRL is a Deep Reinforcement Learning-based tool designed to generate test cases for Lane-Keeping Assist Systems. In this paper, we briefly presents GenRL, and summarize the results of its participation in the Cyber-Physical Systems (CPS) tool competition at SBST 2022.","PeriodicalId":206275,"journal":{"name":"2022 IEEE/ACM 15th International Workshop on Search-Based Software Testing (SBST)","volume":"11 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-05-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"126404491","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
EvoSuite is a search-based unit test generation tool for Java. This paper summarises the results and experiences of EvoSuite's participation at the 10th unit testing competition at SBST 2022, where EvoSuite achieved the highest overall score.
{"title":"EvoSuite at the SBST 2022 Tool Competition","authors":"Sebastian Schweikl, G. Fraser, Andrea Arcuri","doi":"10.1145/3526072.3527526","DOIUrl":"https://doi.org/10.1145/3526072.3527526","url":null,"abstract":"EvoSuite is a search-based unit test generation tool for Java. This paper summarises the results and experiences of EvoSuite's participation at the 10th unit testing competition at SBST 2022, where EvoSuite achieved the highest overall score.","PeriodicalId":206275,"journal":{"name":"2022 IEEE/ACM 15th International Workshop on Search-Based Software Testing (SBST)","volume":"34 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-05-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"133238928","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Basic Block Coverage (BBC) is a secondary objective for search-based unit test generation techniques relying on the approach level and branch distance to drive the search process. Unlike the approach level and branch distance, which considers only information related to the coverage of explicit branches coming from conditional and loop statements, BBC also takes into account implicit branchings (e.g., a runtime exception thrown in a branchless method) denoted by the coverage level of relevant basic blocks in a control flow graph to drive the search process. Our implementation of BBC for unit test generation relies on the DynaMOSA algorithm and EvoSuite. This paper summarizes the results achieved by EvoSuite's DynaMOSA implementation with BBC as a secondary objective at the SBST 2022 unit testing tool competition.
{"title":"Basic Block Coverage for Unit Test Generation at the SBST 2022 Tool Competition","authors":"P. Derakhshanfar, Xavier Devroey","doi":"10.1145/3526072.3527528","DOIUrl":"https://doi.org/10.1145/3526072.3527528","url":null,"abstract":"Basic Block Coverage (BBC) is a secondary objective for search-based unit test generation techniques relying on the approach level and branch distance to drive the search process. Unlike the approach level and branch distance, which considers only information related to the coverage of explicit branches coming from conditional and loop statements, BBC also takes into account implicit branchings (e.g., a runtime exception thrown in a branchless method) denoted by the coverage level of relevant basic blocks in a control flow graph to drive the search process. Our implementation of BBC for unit test generation relies on the DynaMOSA algorithm and EvoSuite. This paper summarizes the results achieved by EvoSuite's DynaMOSA implementation with BBC as a secondary objective at the SBST 2022 unit testing tool competition.","PeriodicalId":206275,"journal":{"name":"2022 IEEE/ACM 15th International Workshop on Search-Based Software Testing (SBST)","volume":"34 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-05-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"134220538","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Kex is an automatic white-box test generation tool for Java programs, which is able to generate executable test suites (as JUnit test suites) aiming to satisfy the branch coverage criterion. It uses symbolic execution to analyze control flow graphs of the program under test (PUT) and produces interesting symbolic inputs for each basic block of PUT. Kex then feeds these inputs to an original backward-search based algorithm called Reanimator, which generates executable JUnit test cases satisfying the symbolic inputs. Kex-reflection is a modification of Kex that uses Java reflection library to generate test cases from symbolic inputs. This paper summarizes the results and experiences of Kex and Kex-reflection participation in the tenth edition of the Java unit testing tool competition at the International Workshop on Search-Based Software Testing (SBST) 2022.
{"title":"Kex at the 2022 SBST Tool Competition","authors":"A. Abdullin, M. Akhin, Mikhail Beliaev","doi":"10.1145/3526072.3527527","DOIUrl":"https://doi.org/10.1145/3526072.3527527","url":null,"abstract":"Kex is an automatic white-box test generation tool for Java programs, which is able to generate executable test suites (as JUnit test suites) aiming to satisfy the branch coverage criterion. It uses symbolic execution to analyze control flow graphs of the program under test (PUT) and produces interesting symbolic inputs for each basic block of PUT. Kex then feeds these inputs to an original backward-search based algorithm called Reanimator, which generates executable JUnit test cases satisfying the symbolic inputs. Kex-reflection is a modification of Kex that uses Java reflection library to generate test cases from symbolic inputs. This paper summarizes the results and experiences of Kex and Kex-reflection participation in the tenth edition of the Java unit testing tool competition at the International Workshop on Search-Based Software Testing (SBST) 2022.","PeriodicalId":206275,"journal":{"name":"2022 IEEE/ACM 15th International Workshop on Search-Based Software Testing (SBST)","volume":"21 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-05-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"114308821","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
WOGAN is an online test generation algorithm based on Wasser-stein generative adversarial networks. In this note, we present how WOGAN works and summarize its performance in the SBST 2022 CPS tool competition concerning the AI of a self-driving car.
{"title":"WOGAN at the SBST 2022 CPS Tool Competition","authors":"J. Peltomäki, Frankie Spencer, Ivan Porres","doi":"10.1145/3526072.3527535","DOIUrl":"https://doi.org/10.1145/3526072.3527535","url":null,"abstract":"WOGAN is an online test generation algorithm based on Wasser-stein generative adversarial networks. In this note, we present how WOGAN works and summarize its performance in the SBST 2022 CPS tool competition concerning the AI of a self-driving car.","PeriodicalId":206275,"journal":{"name":"2022 IEEE/ACM 15th International Workshop on Search-Based Software Testing (SBST)","volume":"100 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-05-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"122320907","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Writing software tests is essential to ensure a high quality of the software project under test. However, writing tests manually is time consuming and expensive. Especially in research fields of the natural sciences, scientists do not have a formal education in software engineering. Thus, automatic test case generation is particularly promising to help build good test suites. In this case study, we investigate the efficacy of automated test case generation approaches for the Python project Atomic Simulation Environment (ASE) used in the material sciences. We compare the branch and mutation coverages reached by both the automatic approaches, as well as a manually created test suite. Finally, we statistically evaluate the measured coverages by each approach against those reached by any of the other approaches. We find that while all evaluated approaches are able to improve upon the original test suite of ASE, none of the automated test case generation algorithms manage to come close to the coverages reached by the manually created test suite. We hypothesize this may be due to the fact that none of the employed test case generation approaches were developed to work on complex structured inputs. Thus, we conclude that more work may be needed if automated test case generation is used on software that requires this type of input.
{"title":"A Comparative Evaluation on the Quality of Manual and Automatic Test Case Generation Techniques for Scientific Software - a Case Study of a Python Project for Material Science Workflows","authors":"Daniel Trübenbach, Sebastian Müller, L. Grunske","doi":"10.1145/3526072.3527523","DOIUrl":"https://doi.org/10.1145/3526072.3527523","url":null,"abstract":"Writing software tests is essential to ensure a high quality of the software project under test. However, writing tests manually is time consuming and expensive. Especially in research fields of the natural sciences, scientists do not have a formal education in software engineering. Thus, automatic test case generation is particularly promising to help build good test suites. In this case study, we investigate the efficacy of automated test case generation approaches for the Python project Atomic Simulation Environment (ASE) used in the material sciences. We compare the branch and mutation coverages reached by both the automatic approaches, as well as a manually created test suite. Finally, we statistically evaluate the measured coverages by each approach against those reached by any of the other approaches. We find that while all evaluated approaches are able to improve upon the original test suite of ASE, none of the automated test case generation algorithms manage to come close to the coverages reached by the manually created test suite. We hypothesize this may be due to the fact that none of the employed test case generation approaches were developed to work on complex structured inputs. Thus, we conclude that more work may be needed if automated test case generation is used on software that requires this type of input.","PeriodicalId":206275,"journal":{"name":"2022 IEEE/ACM 15th International Workshop on Search-Based Software Testing (SBST)","volume":"42 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-05-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"114700686","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
AdaFrenetic is a test generation tool for testing Autonomous Driving System (ADS). It extends the genetic algorithm-based testing tool Frenetic by adjusting the road points to reduce the number of invalid test cases. This paper provides a brief overview of the tool and analyzes the results of AdaFrenetic's performance in the Cyber-physical systems (CPS) testing tool competition at SBST 2022.
{"title":"AdaFrenetic at the SBST 2022 Tool Competition","authors":"Songyang Yan, Ming Fan","doi":"10.1145/3526072.3527530","DOIUrl":"https://doi.org/10.1145/3526072.3527530","url":null,"abstract":"AdaFrenetic is a test generation tool for testing Autonomous Driving System (ADS). It extends the genetic algorithm-based testing tool Frenetic by adjusting the road points to reduce the number of invalid test cases. This paper provides a brief overview of the tool and analyzes the results of AdaFrenetic's performance in the Cyber-physical systems (CPS) testing tool competition at SBST 2022.","PeriodicalId":206275,"journal":{"name":"2022 IEEE/ACM 15th International Workshop on Search-Based Software Testing (SBST)","volume":"226 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-05-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"128777581","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
In Continuous Integration (CI) environments, the productivity of software engineers depends strongly on the ability to reduce the round-trip time between code commits and feedback on failed test cases. Test case prioritization is popularly used as an optimization mechanism for ranking tests by their likelihood in revealing failures. However, existing techniques are usually time and resource intensive making them not suitable to be applied within CI cycles. This paper formulates the test case prioritization problem as an online learn-to-rank model using reinforcement learning techniques. Our approach minimizes the testing overhead and continuously adapts to the changing environment as new code and new test cases are added in each CI cycle. We validated our approach on an industrial case study showing that over 95% of the test failures are still reported back to the software engineers while only 40% of the total available test cases are being executed.
{"title":"Learning to Rank for Test Case Prioritization","authors":"Safa Omri, C. Sinz","doi":"10.1145/3526072.3527525","DOIUrl":"https://doi.org/10.1145/3526072.3527525","url":null,"abstract":"In Continuous Integration (CI) environments, the productivity of software engineers depends strongly on the ability to reduce the round-trip time between code commits and feedback on failed test cases. Test case prioritization is popularly used as an optimization mechanism for ranking tests by their likelihood in revealing failures. However, existing techniques are usually time and resource intensive making them not suitable to be applied within CI cycles. This paper formulates the test case prioritization problem as an online learn-to-rank model using reinforcement learning techniques. Our approach minimizes the testing overhead and continuously adapts to the changing environment as new code and new test cases are added in each CI cycle. We validated our approach on an industrial case study showing that over 95% of the test failures are still reported back to the software engineers while only 40% of the total available test cases are being executed.","PeriodicalId":206275,"journal":{"name":"2022 IEEE/ACM 15th International Workshop on Search-Based Software Testing (SBST)","volume":"65 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-05-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"125632618","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
We report on the organization, challenges, and results of the tenth edition of the Java Unit Testing Competition as well as the second edition of the Cyber-Physical Systems (CPS) Testing Competition. Java Unit Testing Competition. Seven tools, i.e., BBC, EvoSuite, Kex, Kex-Reflection, Randoop, UTBot, and UTBot-Mocks, were executed on a benchmark with 65 classes sampled from four open-source Java projects, for two time budgets: 30 and 120 seconds. CPS Testing Tool Competition. Six tools, i.e., AdaFrenetic, AmbieGen, FreneticV, GenRL, EvoMBT and WOGAN competed on testing two driving agents by generating simulation-based tests. We considered one configuration for each test subject and evaluated the tools' effectiveness and efficiency as well as the failure diversity. This paper describes our methodology, the statistical analysis of the results together with the competing tools, and the challenges faced while running the competition experiments.
{"title":"SBST Tool Competition 2022","authors":"Alessio Gambi, Gunel Jahangirova, Vincenzo Riccio, Fiorella Zampetti","doi":"10.1145/3526072.3527538","DOIUrl":"https://doi.org/10.1145/3526072.3527538","url":null,"abstract":"We report on the organization, challenges, and results of the tenth edition of the Java Unit Testing Competition as well as the second edition of the Cyber-Physical Systems (CPS) Testing Competition. Java Unit Testing Competition. Seven tools, i.e., BBC, EvoSuite, Kex, Kex-Reflection, Randoop, UTBot, and UTBot-Mocks, were executed on a benchmark with 65 classes sampled from four open-source Java projects, for two time budgets: 30 and 120 seconds. CPS Testing Tool Competition. Six tools, i.e., AdaFrenetic, AmbieGen, FreneticV, GenRL, EvoMBT and WOGAN competed on testing two driving agents by generating simulation-based tests. We considered one configuration for each test subject and evaluated the tools' effectiveness and efficiency as well as the failure diversity. This paper describes our methodology, the statistical analysis of the results together with the competing tools, and the challenges faced while running the competition experiments.","PeriodicalId":206275,"journal":{"name":"2022 IEEE/ACM 15th International Workshop on Search-Based Software Testing (SBST)","volume":"30 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-05-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"132613421","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
京公网安备 11010802042870号
京ICP备2023020795号-1
扫码关注我们
求助内容：
应助结果提醒方式：