This paper proposes a measurement framework for assessing the relative quality of alternative approaches to system level model-based testing. The motivation is to investigate the types of measures that the MBT community should apply. The purpose of this paper is to provide a basis for discussion by proposing some initial ideas on where we should probe for MBT quality measurement. The centerpiece of the proposal offered here is the concept of an operational profile (OP) and its relevance to model-based testing.
{"title":"A Measurement Framework for Assessing Model-Based Testing Quality","authors":"P. Lakey","doi":"10.1109/ICSTW.2010.48","DOIUrl":"https://doi.org/10.1109/ICSTW.2010.48","url":null,"abstract":"This paper proposes a measurement framework for assessing the relative quality of alternative approaches to system level model-based testing. The motivation is to investigate the types of measures that the MBT community should apply. The purpose of this paper is to provide a basis for discussion by proposing some initial ideas on where we should probe for MBT quality measurement. The centerpiece of the proposal offered here is the concept of an operational profile (OP) and its relevance to model-based testing.","PeriodicalId":117410,"journal":{"name":"2010 Third International Conference on Software Testing, Verification, and Validation Workshops","volume":"189 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2010-04-06","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"117330226","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Search-based testing techniques using meta-heuristics, like evolutionary algorithms, has been largely used for test data generation, but most approaches were proposed for white-box testing. In this paper we present an evolutionary approach for test sequence generation from a behavior model, in particular, Extended Finite State Machine. An open problem is the production of infeasible paths, as these should be detected and discarded manually. To circumvent this problem, we use an executable model to obtain feasible paths dynamically. An evolutionary algorithm is used to search for solutions that cover a given test purpose, which is a transition of interest. The target transition is used as a criterion to get slicing information, in this way, helping to identify the parts of the model that affect the test purpose. We also present a multi-objective search: the test purpose coverage and the sequence size minimization, as longer sequences require more effort to be executed.
{"title":"Generating Feasible Test Paths from an Executable Model Using a Multi-objective Approach","authors":"T. Yano, E. Martins, F. Sousa","doi":"10.1109/ICSTW.2010.52","DOIUrl":"https://doi.org/10.1109/ICSTW.2010.52","url":null,"abstract":"Search-based testing techniques using meta-heuristics, like evolutionary algorithms, has been largely used for test data generation, but most approaches were proposed for white-box testing. In this paper we present an evolutionary approach for test sequence generation from a behavior model, in particular, Extended Finite State Machine. An open problem is the production of infeasible paths, as these should be detected and discarded manually. To circumvent this problem, we use an executable model to obtain feasible paths dynamically. An evolutionary algorithm is used to search for solutions that cover a given test purpose, which is a transition of interest. The target transition is used as a criterion to get slicing information, in this way, helping to identify the parts of the model that affect the test purpose. We also present a multi-objective search: the test purpose coverage and the sequence size minimization, as longer sequences require more effort to be executed.","PeriodicalId":117410,"journal":{"name":"2010 Third International Conference on Software Testing, Verification, and Validation Workshops","volume":"18 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2010-04-06","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"128474993","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
T. Hanawa, Takayuki Banzai, Hitoshi Koizumi, Ryo Kanbayashi, Takayuki Imada, M. Sato
Various information systems are widely used in information society era, and the demand for highly dependable system is increasing year after year. However, software testing for such a system becomes more difficult due to the enlargement and the complexity of the system. In particular, it is too difficult to test parallel and distributed systems sufficiently although dependable systems such as high-availability servers usually form parallel and distributed systems. To solve these problems, we proposed a software testing environment for dependable parallel and distributed system using the cloud computing technology, named D-Cloud. D-Cloud includes Eucalyptus as the cloud management software, and FaultVM based on QEMU as the virtualization software, and D-Cloud frontend for interpreting test scenario. D-Cloud enables not only to automate the system configuration and the test procedure but also to perform a number of test cases simultaneously, and to emulate hardware faults flexibly. In this paper, we present the concept and design of D-Cloud, and describe how to specify the system configuration and the test scenario. Furthermore, the preliminary test example as the software testing using D-Cloud was presented. Its result shows that D-Cloud allows to set up the environment easily, and to test the software testing for the distributed system.
{"title":"Large-Scale Software Testing Environment Using Cloud Computing Technology for Dependable Parallel and Distributed Systems","authors":"T. Hanawa, Takayuki Banzai, Hitoshi Koizumi, Ryo Kanbayashi, Takayuki Imada, M. Sato","doi":"10.1109/ICSTW.2010.59","DOIUrl":"https://doi.org/10.1109/ICSTW.2010.59","url":null,"abstract":"Various information systems are widely used in information society era, and the demand for highly dependable system is increasing year after year. However, software testing for such a system becomes more difficult due to the enlargement and the complexity of the system. In particular, it is too difficult to test parallel and distributed systems sufficiently although dependable systems such as high-availability servers usually form parallel and distributed systems. To solve these problems, we proposed a software testing environment for dependable parallel and distributed system using the cloud computing technology, named D-Cloud. D-Cloud includes Eucalyptus as the cloud management software, and FaultVM based on QEMU as the virtualization software, and D-Cloud frontend for interpreting test scenario. D-Cloud enables not only to automate the system configuration and the test procedure but also to perform a number of test cases simultaneously, and to emulate hardware faults flexibly. In this paper, we present the concept and design of D-Cloud, and describe how to specify the system configuration and the test scenario. Furthermore, the preliminary test example as the software testing using D-Cloud was presented. Its result shows that D-Cloud allows to set up the environment easily, and to test the software testing for the distributed system.","PeriodicalId":117410,"journal":{"name":"2010 Third International Conference on Software Testing, Verification, and Validation Workshops","volume":"25 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2010-04-06","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"114767530","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
V. Jagannath, Miloš Gligorić, Steven Lauterburg, D. Marinov, G. Agha
Mutation testing is a well known technique for estimating and improving the quality of test suites. Given a test suite T for a system S, mutation testing systematically creates mutants of S and executes T to measure how many mutants T detects. If T does not detect some (non-equivalent) mutants, T can be improved by adding test cases that detect those mutants. Mutants are created by applying mutation operators. Mutation operators are important because they define the characteristics of the system that are tested as well as the characteristics that are improved in the test suite. While mutation operators are well defined for a number of programming paradigms such as sequential or multi-threaded, to the best of our knowledge, mutation operators have not been defined for the actor programming model. In this paper, we define and classify mutation operators that can be used for mutation testing of actor programs.
{"title":"Mutation Operators for Actor Systems","authors":"V. Jagannath, Miloš Gligorić, Steven Lauterburg, D. Marinov, G. Agha","doi":"10.1109/ICSTW.2010.6","DOIUrl":"https://doi.org/10.1109/ICSTW.2010.6","url":null,"abstract":"Mutation testing is a well known technique for estimating and improving the quality of test suites. Given a test suite T for a system S, mutation testing systematically creates mutants of S and executes T to measure how many mutants T detects. If T does not detect some (non-equivalent) mutants, T can be improved by adding test cases that detect those mutants. Mutants are created by applying mutation operators. Mutation operators are important because they define the characteristics of the system that are tested as well as the characteristics that are improved in the test suite. While mutation operators are well defined for a number of programming paradigms such as sequential or multi-threaded, to the best of our knowledge, mutation operators have not been defined for the actor programming model. In this paper, we define and classify mutation operators that can be used for mutation testing of actor programs.","PeriodicalId":117410,"journal":{"name":"2010 Third International Conference on Software Testing, Verification, and Validation Workshops","volume":"28 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2010-04-06","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"133382031","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
The cloud computing model continues to gain much attention from software industry practitioners. As such, leading companies are investing in the development, packaging and delivery of cloud services over the Internet. However, although much work is being done to model and build cloud applications and services, there is significantly less research devoted to testing them. In this paper, we describe our research-in-progress towards migrating autonomic self-testing (AST) to the cloud. Our approach combines the development of an automated test harness for a cloud service, with the delivery of test support as-a-service (TSaaS). Both AST and TSaaS are supported by a virtual test environment, which utilizes the power of the cloud to enhance the self-testing process.
{"title":"Migrating Autonomic Self-Testing to the Cloud","authors":"Tariq M. King, Annaji Sharma Ganti","doi":"10.1109/ICSTW.2010.66","DOIUrl":"https://doi.org/10.1109/ICSTW.2010.66","url":null,"abstract":"The cloud computing model continues to gain much attention from software industry practitioners. As such, leading companies are investing in the development, packaging and delivery of cloud services over the Internet. However, although much work is being done to model and build cloud applications and services, there is significantly less research devoted to testing them. In this paper, we describe our research-in-progress towards migrating autonomic self-testing (AST) to the cloud. Our approach combines the development of an automated test harness for a cloud service, with the delivery of test support as-a-service (TSaaS). Both AST and TSaaS are supported by a virtual test environment, which utilizes the power of the cloud to enhance the self-testing process.","PeriodicalId":117410,"journal":{"name":"2010 Third International Conference on Software Testing, Verification, and Validation Workshops","volume":"16 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2010-04-06","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"133886507","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
In the literature of model-based testing (MBT), different testing scenarios are introduced. These scenarios mainly differ in the origin and content of test models which has an impact on testing activities. For example, if developers models are directly used as test models, the efforts for test model creation will be low, however the evaluation of test results will need more efforts [2]. We investigated the differences of MBT scenarios from a test manager's point of view trying to answer the following questions: Which efforts must be calculated when applying the different MBT scenarios? Can we systematically analyze and compare the different scenarios in the literature? For answering these questions we propose a GQM-based approach where we systematically define metrics for characterizing and comparing MBT scenarios.
{"title":"Effort Comparison for Model-Based Testing Scenarios","authors":"Baris Güldali, M. Mlynarski, Yavuz Sancar","doi":"10.1109/ICSTW.2010.15","DOIUrl":"https://doi.org/10.1109/ICSTW.2010.15","url":null,"abstract":"In the literature of model-based testing (MBT), different testing scenarios are introduced. These scenarios mainly differ in the origin and content of test models which has an impact on testing activities. For example, if developers models are directly used as test models, the efforts for test model creation will be low, however the evaluation of test results will need more efforts [2]. We investigated the differences of MBT scenarios from a test manager's point of view trying to answer the following questions: Which efforts must be calculated when applying the different MBT scenarios? Can we systematically analyze and compare the different scenarios in the literature? For answering these questions we propose a GQM-based approach where we systematically define metrics for characterizing and comparing MBT scenarios.","PeriodicalId":117410,"journal":{"name":"2010 Third International Conference on Software Testing, Verification, and Validation Workshops","volume":"9 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2010-04-06","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"115352692","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
New coverage criteria for Boolean expressions are regularly introduced with two goals: to detect specific classes of realistic faults and to produce as small as possible test suites. In this paper we investigate whether an approach targeting specific fault classes using several reduction policies can achieve that less test cases are generated than by previously introduced testing criteria. In our approach, the problem of finding fault detecting test cases can be formalized as a logical satisfiability problem, which can be efficiently solved by a SAT algorithm. We compare this approach with respect to the well-known MUMCUT and Minimal-MUMCUT strategies by applying it to a series of case studies commonly used as benchmarks, and show that it can reduce the number of test cases further than Minimal-MUMCUT.
{"title":"Generating Minimal Fault Detecting Test Suites for Boolean Expressions","authors":"G. Fraser, A. Gargantini","doi":"10.1109/ICSTW.2010.51","DOIUrl":"https://doi.org/10.1109/ICSTW.2010.51","url":null,"abstract":"New coverage criteria for Boolean expressions are regularly introduced with two goals: to detect specific classes of realistic faults and to produce as small as possible test suites. In this paper we investigate whether an approach targeting specific fault classes using several reduction policies can achieve that less test cases are generated than by previously introduced testing criteria. In our approach, the problem of finding fault detecting test cases can be formalized as a logical satisfiability problem, which can be efficiently solved by a SAT algorithm. We compare this approach with respect to the well-known MUMCUT and Minimal-MUMCUT strategies by applying it to a series of case studies commonly used as benchmarks, and show that it can reduce the number of test cases further than Minimal-MUMCUT.","PeriodicalId":117410,"journal":{"name":"2010 Third International Conference on Software Testing, Verification, and Validation Workshops","volume":"37 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2010-04-06","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"116132258","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
In industrial practice, developers are often unable to reproduce errors that are encountered by end-users or testers. Evidently, reproducibility is important for investigating the root cause of the error, since without knowing what causes the error, a developer cannot repair the software. This paper reports on the successful application of evolutionary testing by Rila Solution EAD to solve two real reproducibility problems they encountered. Rila's software application, the ChatPC, suffered from memory and data corruption faults, that were reported by users in the field but could not be reproduced in-house after considerable effort. This paper presents two case studies that show how evolutionary testing resulted in finding execution scenarios that could reliably reproduce the mentioned faults observed in the application.
{"title":"Using Evolutionary Testing to Find Test Scenarios for Hard to Reproduce Faults","authors":"A. Baars, T. Vos, Dimitar M. Dimitrov","doi":"10.1109/ICSTW.2010.30","DOIUrl":"https://doi.org/10.1109/ICSTW.2010.30","url":null,"abstract":"In industrial practice, developers are often unable to reproduce errors that are encountered by end-users or testers. Evidently, reproducibility is important for investigating the root cause of the error, since without knowing what causes the error, a developer cannot repair the software. This paper reports on the successful application of evolutionary testing by Rila Solution EAD to solve two real reproducibility problems they encountered. Rila's software application, the ChatPC, suffered from memory and data corruption faults, that were reported by users in the field but could not be reproduced in-house after considerable effort. This paper presents two case studies that show how evolutionary testing resulted in finding execution scenarios that could reliably reproduce the mentioned faults observed in the application.","PeriodicalId":117410,"journal":{"name":"2010 Third International Conference on Software Testing, Verification, and Validation Workshops","volume":"42 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2010-04-06","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"131644404","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Today GUIs are not exclusive of desktops and web applications. They can be found in a widely variety of embedded systems such as cellular phones, TVs, cars, etc. GUI testing is an emergent approach to assure software quality. In this paper, we show how to evaluate some GUI testing techniques and the importance of controlled experiments in order to have statistical confidence. Furthermore, as GUI design changes often during the development process, test cases need to be updated as well. Therefore, we also propose a general framework to GUI test case design and generation based on model-based testing and GUI capture-replay tools. The framework is easily extended to support different test case generation algorithms and script languages. It also allows one to explore functional as well as non-functional requirements, such as usability, accessibility, reliability, and so on.
{"title":"A Framework for GUI Testing Based on Use Case Design","authors":"C. Bertolini, A. Mota","doi":"10.1109/ICSTW.2010.37","DOIUrl":"https://doi.org/10.1109/ICSTW.2010.37","url":null,"abstract":"Today GUIs are not exclusive of desktops and web applications. They can be found in a widely variety of embedded systems such as cellular phones, TVs, cars, etc. GUI testing is an emergent approach to assure software quality. In this paper, we show how to evaluate some GUI testing techniques and the importance of controlled experiments in order to have statistical confidence. Furthermore, as GUI design changes often during the development process, test cases need to be updated as well. Therefore, we also propose a general framework to GUI test case design and generation based on model-based testing and GUI capture-replay tools. The framework is easily extended to support different test case generation algorithms and script languages. It also allows one to explore functional as well as non-functional requirements, such as usability, accessibility, reliability, and so on.","PeriodicalId":117410,"journal":{"name":"2010 Third International Conference on Software Testing, Verification, and Validation Workshops","volume":"1 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2010-04-06","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"129307839","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
In this paper, we investigate the problem of constructing a test case for a given test purpose and specification modelled by input/output transition systems (IOTS). The communication between the tester and the implementation under test is assumed to be asynchronous, performed via queues. Differently from synchronous tests, when issuing verdicts, asynchronous tests should take into account the distortion caused by the queues in the observed interactions. We propose an algorithm which constructs a sound test case, by transforming the test purpose prior to composing it with the specification without queues, mitigating the state explosion problem which usually occurs when queues are directly involved in the composition. Experimental results confirm the resulting state space reduction. The proposed approach can also be used to obtain sound asynchronous tests from synchronous tests. We identify a class of IOTS specifications for which synchronous and asynchronous tests coincide.
{"title":"From Test Purposes to Asynchronous Test Cases","authors":"A. Simão, A. Petrenko","doi":"10.1109/ICSTW.2010.32","DOIUrl":"https://doi.org/10.1109/ICSTW.2010.32","url":null,"abstract":"In this paper, we investigate the problem of constructing a test case for a given test purpose and specification modelled by input/output transition systems (IOTS). The communication between the tester and the implementation under test is assumed to be asynchronous, performed via queues. Differently from synchronous tests, when issuing verdicts, asynchronous tests should take into account the distortion caused by the queues in the observed interactions. We propose an algorithm which constructs a sound test case, by transforming the test purpose prior to composing it with the specification without queues, mitigating the state explosion problem which usually occurs when queues are directly involved in the composition. Experimental results confirm the resulting state space reduction. The proposed approach can also be used to obtain sound asynchronous tests from synchronous tests. We identify a class of IOTS specifications for which synchronous and asynchronous tests coincide.","PeriodicalId":117410,"journal":{"name":"2010 Third International Conference on Software Testing, Verification, and Validation Workshops","volume":"39 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2010-04-06","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"126668259","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
京公网安备 11010802042870号
京ICP备2023020795号-1
扫码关注我们
求助内容：
应助结果提醒方式：