Pub Date : 2022-04-01DOI: 10.1109/ICSTW55395.2022.00048
Wen-Xin Zhang
Coverage-Guided fuzzing is the type of fuzzing which focuses on the code or branch coverage. It is mainly efficient in detecting buffer overflow. For the code coverage, it is an important parameter of performance evaluation of the Coverage-Guided fuzzing tools, since the higher coverage means higher possibility of bug detection. However, the timeout set for the fuzzing also affects the efficiency, due to the fact that the growth rate of code coverage will gradually become slower as the running time becoming longer. Setting a timeout that is too long cannot help improving the coverage nor the bug detected, and it will be a complete waste of time. Since selecting the fuzzing time might be quite confusing for testers, in this paper, the author decided to discover the relationship of the growth of the coverage and the running time of the. AFL and FuzzFactory were used for the evaluation and the timeout were set as 1 second, 1 minute, 1 hour, 6 hours and 12 hours respectively. The results showed the relationship between fuzzing time and the performance, for all fuzzers and the effect between different fuzzers.
{"title":"Obtaining Fuzzing Results with Different Timeouts","authors":"Wen-Xin Zhang","doi":"10.1109/ICSTW55395.2022.00048","DOIUrl":"https://doi.org/10.1109/ICSTW55395.2022.00048","url":null,"abstract":"Coverage-Guided fuzzing is the type of fuzzing which focuses on the code or branch coverage. It is mainly efficient in detecting buffer overflow. For the code coverage, it is an important parameter of performance evaluation of the Coverage-Guided fuzzing tools, since the higher coverage means higher possibility of bug detection. However, the timeout set for the fuzzing also affects the efficiency, due to the fact that the growth rate of code coverage will gradually become slower as the running time becoming longer. Setting a timeout that is too long cannot help improving the coverage nor the bug detected, and it will be a complete waste of time. Since selecting the fuzzing time might be quite confusing for testers, in this paper, the author decided to discover the relationship of the growth of the coverage and the running time of the. AFL and FuzzFactory were used for the evaluation and the timeout were set as 1 second, 1 minute, 1 hour, 6 hours and 12 hours respectively. The results showed the relationship between fuzzing time and the performance, for all fuzzers and the effect between different fuzzers.","PeriodicalId":147133,"journal":{"name":"2022 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)","volume":"19 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-04-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"123118387","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2022-04-01DOI: 10.1109/ICSTW55395.2022.00051
J. Hagar
This paper presents definitions of test architecture views, viewpoints, and containers. The paper encourages debate within the modelling and architecture community and builds on previous work and papers. There are job postings for test architects. However, many testers and system architects do not consider test modeling and planning to include test architectures, views, viewpoints, and contents. The world of standards, particularly test standards, has not yet reached a consensus on software test architectures. This paper also presents definitions and considerations for architectural elements and concepts within a test perspective.
{"title":"Software Architecture Elements Applied to Software Test: View, Viewpoints and Containers","authors":"J. Hagar","doi":"10.1109/ICSTW55395.2022.00051","DOIUrl":"https://doi.org/10.1109/ICSTW55395.2022.00051","url":null,"abstract":"This paper presents definitions of test architecture views, viewpoints, and containers. The paper encourages debate within the modelling and architecture community and builds on previous work and papers. There are job postings for test architects. However, many testers and system architects do not consider test modeling and planning to include test architectures, views, viewpoints, and contents. The world of standards, particularly test standards, has not yet reached a consensus on software test architectures. This paper also presents definitions and considerations for architectural elements and concepts within a test perspective.","PeriodicalId":147133,"journal":{"name":"2022 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)","volume":"325 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-04-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"115761985","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2022-04-01DOI: 10.1109/ICSTW55395.2022.00021
Zujany Salazar, A. Cavalli, Wissam Mallouli, Filip Sebek, Fatiha Zaïdi, M. Rakoczy
Safety monitoring of Industrial Control Systems (ICS) is a must for optimal operation of safe manufacturing facilities. Failures and miss-behaviours seldomly occur without prior warning, but these warnings are often subtle, requiring careful analysis of data by experienced personnel for early detection. Monitoring function allows to promptly take adequate corrective actions in order to maximize uptime and increase trust of running industrial systems. In this paper, we present two main approaches of monitoring techniques implemented in the Montimage MMT tool. The first approach is a signature-based approach, where there are safety properties to be checked on the ICS logs, and the other relies on Machine Learning (ML) to detect anomalies. Both methods have been applied to check safety on an industrial system: a crane load position system provided by ABB. Several experiments have been performed to check if the information provided by a system’s PLC is correct, guarantying the safety of the system.
{"title":"Monitoring Approaches for Security and Safety Analysis: Application to a Load Position System","authors":"Zujany Salazar, A. Cavalli, Wissam Mallouli, Filip Sebek, Fatiha Zaïdi, M. Rakoczy","doi":"10.1109/ICSTW55395.2022.00021","DOIUrl":"https://doi.org/10.1109/ICSTW55395.2022.00021","url":null,"abstract":"Safety monitoring of Industrial Control Systems (ICS) is a must for optimal operation of safe manufacturing facilities. Failures and miss-behaviours seldomly occur without prior warning, but these warnings are often subtle, requiring careful analysis of data by experienced personnel for early detection. Monitoring function allows to promptly take adequate corrective actions in order to maximize uptime and increase trust of running industrial systems. In this paper, we present two main approaches of monitoring techniques implemented in the Montimage MMT tool. The first approach is a signature-based approach, where there are safety properties to be checked on the ICS logs, and the other relies on Machine Learning (ML) to detect anomalies. Both methods have been applied to check safety on an industrial system: a crane load position system provided by ABB. Several experiments have been performed to check if the information provided by a system’s PLC is correct, guarantying the safety of the system.","PeriodicalId":147133,"journal":{"name":"2022 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)","volume":"7 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-04-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"121927060","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2022-04-01DOI: 10.1109/icstw55395.2022.00007
{"title":"Message from the AIST 2022 Chair","authors":"","doi":"10.1109/icstw55395.2022.00007","DOIUrl":"https://doi.org/10.1109/icstw55395.2022.00007","url":null,"abstract":"","PeriodicalId":147133,"journal":{"name":"2022 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)","volume":"38 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-04-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"124837583","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2022-04-01DOI: 10.1109/ICSTW55395.2022.00022
Rui Huang, Chang Rao, Yu Lei, Jin Guo, Yadong Zhang
The onboard Automatic Train Protection System (ATP) is one of the key components of the Chinese high-speed railway train control system. ATP is a safety-critical system since a failure of ATP could result in serious accidents. This paper reports a combinatorial testing practice performed in testing one of the major ATP functions, i.e. Balise Information Processing (BIP). We created one input model for each of the total 7 application scenarios of BIP. We generated a total of 178 pair-wise tests using the ACTS tool. We executed all these 178 tests, among which 172 tests passed and 6 tests failed. We found a total of 5 new faults, including 2 critical faults, and 3 major faults. We believe that combinatorial testing can be a very effective approach to testing large and complex real-world systems such as ATP.
{"title":"Applying Combinatorial Testing to High-Speed Railway Automatic Train Protection System","authors":"Rui Huang, Chang Rao, Yu Lei, Jin Guo, Yadong Zhang","doi":"10.1109/ICSTW55395.2022.00022","DOIUrl":"https://doi.org/10.1109/ICSTW55395.2022.00022","url":null,"abstract":"The onboard Automatic Train Protection System (ATP) is one of the key components of the Chinese high-speed railway train control system. ATP is a safety-critical system since a failure of ATP could result in serious accidents. This paper reports a combinatorial testing practice performed in testing one of the major ATP functions, i.e. Balise Information Processing (BIP). We created one input model for each of the total 7 application scenarios of BIP. We generated a total of 178 pair-wise tests using the ACTS tool. We executed all these 178 tests, among which 172 tests passed and 6 tests failed. We found a total of 5 new faults, including 2 critical faults, and 3 major faults. We believe that combinatorial testing can be a very effective approach to testing large and complex real-world systems such as ATP.","PeriodicalId":147133,"journal":{"name":"2022 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)","volume":"25 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-04-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"134036226","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2022-04-01DOI: 10.1109/ICSTW55395.2022.00058
Aftab Hussain, Mohammad Amin Alipour
Software fuzzing mutates bytes in test seeds to explore different behaviors of a program under test. Initial seeds can have great impact on the performance of fuzzing campaigns. Mutating a lot of uninteresting bytes in a large seed wastes the fuzzing resources and slows down the exploration of important parts of the program. However, identifying "uninteresting" bytes is difficult. In this paper, we propose and evaluate Diar, a simple approach for mitigating the problem of uninteresting bytes in the seeds. In this approach, we call a byte uninteresting if its removal does not substantially change the coverage of a seed. Next, we use the non-adequate test reduction technique to remove such bytes in the seeds. We performed a preliminary study by applying this approach on the initial seeds in two fuzzing campaigns. Our results suggest fuzzing campaigns that start with reduced seeds, find new paths faster, and can produce higher coverage overall.
{"title":"Removing Uninteresting Bytes in Software Fuzzing","authors":"Aftab Hussain, Mohammad Amin Alipour","doi":"10.1109/ICSTW55395.2022.00058","DOIUrl":"https://doi.org/10.1109/ICSTW55395.2022.00058","url":null,"abstract":"Software fuzzing mutates bytes in test seeds to explore different behaviors of a program under test. Initial seeds can have great impact on the performance of fuzzing campaigns. Mutating a lot of uninteresting bytes in a large seed wastes the fuzzing resources and slows down the exploration of important parts of the program. However, identifying \"uninteresting\" bytes is difficult. In this paper, we propose and evaluate Diar, a simple approach for mitigating the problem of uninteresting bytes in the seeds. In this approach, we call a byte uninteresting if its removal does not substantially change the coverage of a seed. Next, we use the non-adequate test reduction technique to remove such bytes in the seeds. We performed a preliminary study by applying this approach on the initial seeds in two fuzzing campaigns. Our results suggest fuzzing campaigns that start with reduced seeds, find new paths faster, and can produce higher coverage overall.","PeriodicalId":147133,"journal":{"name":"2022 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)","volume":"22 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-04-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"134539620","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2022-04-01DOI: 10.1109/ICSTW55395.2022.00036
Vahana Dorcis, F. Bouquet, Frédéric Dadeau
Our objective is to define a regression testing approach that relies on usage traces that capture the behaviours of the system when exploited by the users. We achieve that by studying and evaluating clustering techniques applied to usage traces for regression tests selection. We first evaluate the existing vectorization methods and the clusters computed by the classical algorithms, and then, evaluate the clusters using existing state-of-the-art validation methods. We conclude that neither the existing clustering algorithms, nor the seminal clustering evaluation techniques are well-suited for identifying representative behaviours of the system when applied to usage traces. Thus, we propose a custom clustering algorithm and a dedicated cluster evaluation index for selecting usage trace to be used as regression tests.
{"title":"Clustering of Usage Traces for Regression Test Cases Selection","authors":"Vahana Dorcis, F. Bouquet, Frédéric Dadeau","doi":"10.1109/ICSTW55395.2022.00036","DOIUrl":"https://doi.org/10.1109/ICSTW55395.2022.00036","url":null,"abstract":"Our objective is to define a regression testing approach that relies on usage traces that capture the behaviours of the system when exploited by the users. We achieve that by studying and evaluating clustering techniques applied to usage traces for regression tests selection. We first evaluate the existing vectorization methods and the clusters computed by the classical algorithms, and then, evaluate the clusters using existing state-of-the-art validation methods. We conclude that neither the existing clustering algorithms, nor the seminal clustering evaluation techniques are well-suited for identifying representative behaviours of the system when applied to usage traces. Thus, we propose a custom clustering algorithm and a dedicated cluster evaluation index for selecting usage trace to be used as regression tests.","PeriodicalId":147133,"journal":{"name":"2022 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)","volume":"26 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-04-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"133864988","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2022-04-01DOI: 10.1109/ICSTW55395.2022.00018
J. Peltomäki, Ivan Porres
We consider the problem of falsifying safety requirements of Cyber-Physical Systems expressed in signal temporal logic (STL). This problem can be turned into an optimization problem via STL robustness functions. In this paper, our focus is in falsifying systems with multiple requirements. We propose to solve such conjunctive requirements using online generative adversarial networks (GANs) as test generators. Our main contribution is an algorithm which falsifies a conjunctive requirement φ1 ∧•⋯•∧φn by using a GAN for each requirement φi separately. Using ideas from multi-armed bandit algorithms, our algorithm only trains a single GAN at every step, which saves resources. Our experiments indicate that, in addition to saving resources, this multi-armed bandit algorithm can falsify requirements with fewer number of executions on the system under test when compared to (i) an algorithm training a single GAN for the complete conjunctive requirement and (ii) an algorithm always training n GANs at each step.
{"title":"Falsification of Multiple Requirements for Cyber-Physical Systems Using Online Generative Adversarial Networks and Multi-Armed Bandits","authors":"J. Peltomäki, Ivan Porres","doi":"10.1109/ICSTW55395.2022.00018","DOIUrl":"https://doi.org/10.1109/ICSTW55395.2022.00018","url":null,"abstract":"We consider the problem of falsifying safety requirements of Cyber-Physical Systems expressed in signal temporal logic (STL). This problem can be turned into an optimization problem via STL robustness functions. In this paper, our focus is in falsifying systems with multiple requirements. We propose to solve such conjunctive requirements using online generative adversarial networks (GANs) as test generators. Our main contribution is an algorithm which falsifies a conjunctive requirement φ1 ∧•⋯•∧φn by using a GAN for each requirement φi separately. Using ideas from multi-armed bandit algorithms, our algorithm only trains a single GAN at every step, which saves resources. Our experiments indicate that, in addition to saving resources, this multi-armed bandit algorithm can falsify requirements with fewer number of executions on the system under test when compared to (i) an algorithm training a single GAN for the complete conjunctive requirement and (ii) an algorithm always training n GANs at each step.","PeriodicalId":147133,"journal":{"name":"2022 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)","volume":"335 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-04-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"115843525","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2022-04-01DOI: 10.1109/ICSTW55395.2022.00014
Markus Borg
This talk shares lessons learned from using search-based techniques for robustness testing in simulators.
本次演讲将分享在模拟器中使用基于搜索的技术进行鲁棒性测试的经验教训。
{"title":"Using Search-Based Software Testing to Guide the Strive for Robust Machine Learning Components: Lessons Learned Across Systems and Simulators in the Mobility Domain","authors":"Markus Borg","doi":"10.1109/ICSTW55395.2022.00014","DOIUrl":"https://doi.org/10.1109/ICSTW55395.2022.00014","url":null,"abstract":"This talk shares lessons learned from using search-based techniques for robustness testing in simulators.","PeriodicalId":147133,"journal":{"name":"2022 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)","volume":"478 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-04-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"115950123","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2022-04-01DOI: 10.1109/ICSTW55395.2022.00053
Felix Schuckert, Hanno Langweg, Basel Katt
Synthetic static code analysis test suites are important to test the basic functionality of tools. We present a framework that uses different source code patterns to generate Cross Site Scripting and SQL injection test cases. A decision tree is used to determine if the test cases are vulnerable. The test cases are split into two test suites. The first test suite contains 258,432 test cases that have influence on the decision trees. The second test suite contains 20 vulnerable test cases with different data flow patterns. The test cases are scanned with two commercial static code analysis tools to show that they can be used to benchmark and identify problems of static code analysis tools. Expert interviews confirm that the decision tree is a solid way to determine the vulnerable test cases and that the test suites are relevant.
{"title":"Systematic Generation of XSS and SQLi Vulnerabilities in PHP as Test Cases for Static Code Analysis","authors":"Felix Schuckert, Hanno Langweg, Basel Katt","doi":"10.1109/ICSTW55395.2022.00053","DOIUrl":"https://doi.org/10.1109/ICSTW55395.2022.00053","url":null,"abstract":"Synthetic static code analysis test suites are important to test the basic functionality of tools. We present a framework that uses different source code patterns to generate Cross Site Scripting and SQL injection test cases. A decision tree is used to determine if the test cases are vulnerable. The test cases are split into two test suites. The first test suite contains 258,432 test cases that have influence on the decision trees. The second test suite contains 20 vulnerable test cases with different data flow patterns. The test cases are scanned with two commercial static code analysis tools to show that they can be used to benchmark and identify problems of static code analysis tools. Expert interviews confirm that the decision tree is a solid way to determine the vulnerable test cases and that the test suites are relevant.","PeriodicalId":147133,"journal":{"name":"2022 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)","volume":"44 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-04-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"116352079","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
京公网安备 11010802042870号
京ICP备2023020795号-1
扫码关注我们
求助内容：
应助结果提醒方式：