Recent code-based test input generators based on dynamic symbolic execution increase path coverage by solving path condition with a constraint or an SMT solver. When the solver considers path condition produced from an infeasible path, it tries to show unsatisfiability, which is a useless time-consuming process. In this paper, we propose a new method that takes opportunity of the detection of a single infeasible path to generalize to a (possibly infinite) family of infeasible paths, which will not have to be considered in further path conditions solving. The method exploits non-intrusive constraint-based explanations, a technique developed in Constraint Programming to explain unsatisfiability. Experimental results obtained with our prototype tool IPEG show that, whatever is the underlying constraint solving procedure (IC, Colibri and the SMT solver Z3), this approach can save considerable computational time.
{"title":"Explanation-Based Generalization of Infeasible Path","authors":"Mickaël Delahaye, Bernard Botella, A. Gotlieb","doi":"10.1109/ICST.2010.13","DOIUrl":"https://doi.org/10.1109/ICST.2010.13","url":null,"abstract":"Recent code-based test input generators based on dynamic symbolic execution increase path coverage by solving path condition with a constraint or an SMT solver. When the solver considers path condition produced from an infeasible path, it tries to show unsatisfiability, which is a useless time-consuming process. In this paper, we propose a new method that takes opportunity of the detection of a single infeasible path to generalize to a (possibly infinite) family of infeasible paths, which will not have to be considered in further path conditions solving. The method exploits non-intrusive constraint-based explanations, a technique developed in Constraint Programming to explain unsatisfiability. Experimental results obtained with our prototype tool IPEG show that, whatever is the underlying constraint solving procedure (IC, Colibri and the SMT solver Z3), this approach can save considerable computational time.","PeriodicalId":192678,"journal":{"name":"2010 Third International Conference on Software Testing, Verification and Validation","volume":"39 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2010-04-06","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"126107657","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Verification of models and model processing programs are inevitable in real-world model-based software development. Model transformation developers are interested in offline verification methods, when only the definition of the model transformation and the metamodels of the source and target languages are used to analyze the properties and no concrete input models are taken into account. Therefore, the results of the analysis hold for each output model not just particular ones, and we have to perform the analysis only once. Most often, formal verification of model transformations is performed manually or the methods can be applied only for a certain transformation or for the analysis of only a certain property. Previous work has presented a formalism to describe the characteristics of model transformations in separate formal expressions called assertions. This description is based on the first-order logic, therefore, if deduction rules are provided, a reasoning system can use an assertion set to automatically derive additional assertions describing additional properties of model transformations. In this paper, we propose deduction rules and present the verification of a model transformation of processing business process models.
{"title":"Towards Automated, Formal Verification of Model Transformations","authors":"M. Asztalos, L. Lengyel, T. Levendovszky","doi":"10.1109/ICST.2010.42","DOIUrl":"https://doi.org/10.1109/ICST.2010.42","url":null,"abstract":"Verification of models and model processing programs are inevitable in real-world model-based software development. Model transformation developers are interested in offline verification methods, when only the definition of the model transformation and the metamodels of the source and target languages are used to analyze the properties and no concrete input models are taken into account. Therefore, the results of the analysis hold for each output model not just particular ones, and we have to perform the analysis only once. Most often, formal verification of model transformations is performed manually or the methods can be applied only for a certain transformation or for the analysis of only a certain property. Previous work has presented a formalism to describe the characteristics of model transformations in separate formal expressions called assertions. This description is based on the first-order logic, therefore, if deduction rules are provided, a reasoning system can use an assertion set to automatically derive additional assertions describing additional properties of model transformations. In this paper, we propose deduction rules and present the verification of a model transformation of processing business process models.","PeriodicalId":192678,"journal":{"name":"2010 Third International Conference on Software Testing, Verification and Validation","volume":"63 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2010-04-06","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"114397195","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
This paper introduces the research topic of “Elaborating Software Test Processes and Strategies”, in which I with assistance from our research group conducted a qualitative study on software producing organization’s test processes. My goal is to develop a reference model for organizations to enhance and develop their test processes based on the upcoming ISO/IEC 29119 software testing standard by interviewing professional software developers from different phases of software process. The study focuses on the aspects that compose testing strategy; human resources, test tools, test case selection, testing methods and the role of the management in the test process to name few of the major components. Based on the preliminary results, there exists room for improvements in testability of the software products and focusing the available test resources, problems which could be addressed by means of systematic process improvement and defining test strategy for the organization.
{"title":"Elaborating Software Test Processes and Strategies","authors":"Jussi Kasurinen","doi":"10.1109/ICST.2010.25","DOIUrl":"https://doi.org/10.1109/ICST.2010.25","url":null,"abstract":"This paper introduces the research topic of “Elaborating Software Test Processes and Strategies”, in which I with assistance from our research group conducted a qualitative study on software producing organization’s test processes. My goal is to develop a reference model for organizations to enhance and develop their test processes based on the upcoming ISO/IEC 29119 software testing standard by interviewing professional software developers from different phases of software process. The study focuses on the aspects that compose testing strategy; human resources, test tools, test case selection, testing methods and the role of the management in the test process to name few of the major components. Based on the preliminary results, there exists room for improvements in testability of the software products and focusing the available test resources, problems which could be addressed by means of systematic process improvement and defining test strategy for the organization.","PeriodicalId":192678,"journal":{"name":"2010 Third International Conference on Software Testing, Verification and Validation","volume":"113 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2010-04-06","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"129640740","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
F. Belli, Michael Linschulte, Christof J. Budnik, H. Stieber
Testing of graphical user interfaces is important due to its potential to reveal faults in operation and performance of the system under consideration. Most existing test approaches generate test cases as sequences of events of different length. The cost of the test process depends on the number and total length of those test sequences. One of the problems to be encountered is the determination of the test sequence length. Widely accepted hypothesis is that the longer the test sequences, the higher the chances to detect faults. However, there is no evidence that an increase of the test sequence length really affect the fault detection. This paper introduces a reliability theoretical approach to analyze the problem in the light of real-life case studies. Based on a reliability growth model the expected number of additional faults is predicted that will be detected when increasing the length of test sequences.
{"title":"Fault Detection Likelihood of Test Sequence Length","authors":"F. Belli, Michael Linschulte, Christof J. Budnik, H. Stieber","doi":"10.1109/ICST.2010.51","DOIUrl":"https://doi.org/10.1109/ICST.2010.51","url":null,"abstract":"Testing of graphical user interfaces is important due to its potential to reveal faults in operation and performance of the system under consideration. Most existing test approaches generate test cases as sequences of events of different length. The cost of the test process depends on the number and total length of those test sequences. One of the problems to be encountered is the determination of the test sequence length. Widely accepted hypothesis is that the longer the test sequences, the higher the chances to detect faults. However, there is no evidence that an increase of the test sequence length really affect the fault detection. This paper introduces a reliability theoretical approach to analyze the problem in the light of real-life case studies. Based on a reliability growth model the expected number of additional faults is predicted that will be detected when increasing the length of test sequences.","PeriodicalId":192678,"journal":{"name":"2010 Third International Conference on Software Testing, Verification and Validation","volume":"120 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2010-04-06","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"128130883","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Mijung Kim, S. Sinha, C. Görg, Hina B. Shah, M. J. Harrold, M. Nanda
Although many static-analysis techniques have been developed for automatically detecting bugs, such as null dereferences, fewer automated approaches have been presented for analyzing whether and how such bugs are fixed. Attempted bug fixes may be incomplete in that a related manifestation of the bug remains unfixed. In this paper, we characterize the “completeness” of attempted bug fixes that involve the flow of invalid values from one program point to another, such as null dereferences, in Java programs. Our characterization is based on the definition of a bug neighborhood, which is a scope of flows of invalid values. We present an automated analysis that, given two versions P and P' of a program, identifies the bugs in P that have been fixed in P', and classifies each fix as complete or incomplete. We implemented our technique for null-dereference bugs and conducted empirical studies using open-source projects. Our results indicate that, for the projects we studied, many bug fixes are not complete, and thus, may cause failures in subsequent executions of the program.
{"title":"Automated Bug Neighborhood Analysis for Identifying Incomplete Bug Fixes","authors":"Mijung Kim, S. Sinha, C. Görg, Hina B. Shah, M. J. Harrold, M. Nanda","doi":"10.1109/ICST.2010.63","DOIUrl":"https://doi.org/10.1109/ICST.2010.63","url":null,"abstract":"Although many static-analysis techniques have been developed for automatically detecting bugs, such as null dereferences, fewer automated approaches have been presented for analyzing whether and how such bugs are fixed. Attempted bug fixes may be incomplete in that a related manifestation of the bug remains unfixed. In this paper, we characterize the “completeness” of attempted bug fixes that involve the flow of invalid values from one program point to another, such as null dereferences, in Java programs. Our characterization is based on the definition of a bug neighborhood, which is a scope of flows of invalid values. We present an automated analysis that, given two versions P and P' of a program, identifies the bugs in P that have been fixed in P', and classifies each fix as complete or incomplete. We implemented our technique for null-dereference bugs and conducted empirical studies using open-source projects. Our results indicate that, for the projects we studied, many bug fixes are not complete, and thus, may cause failures in subsequent executions of the program.","PeriodicalId":192678,"journal":{"name":"2010 Third International Conference on Software Testing, Verification and Validation","volume":"17 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2010-04-06","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"127843297","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
This paper presents TestFul, an evolutionary testing approach for Java classes that works both at class and method level. TestFul exploits a multi-objective evolutionary algorithm to identify the “best” tests. The paper introduces the main elements of TestFul. It also compares TestFul against well-known search-based solutions using a set of classes taken from literature, known software libraries, and independent testing benchmarks. The comparison considers statement and branch coverage, size of generated tests, and generation time. On considered classes, TestFul generates better tests than other search-based solutions, and achieves higher structural coverages with tests small enough to be usable.
{"title":"TestFul: An Evolutionary Test Approach for Java","authors":"L. Baresi, P. Lanzi, Matteo Miraz","doi":"10.1109/ICST.2010.54","DOIUrl":"https://doi.org/10.1109/ICST.2010.54","url":null,"abstract":"This paper presents TestFul, an evolutionary testing approach for Java classes that works both at class and method level. TestFul exploits a multi-objective evolutionary algorithm to identify the “best” tests. The paper introduces the main elements of TestFul. It also compares TestFul against well-known search-based solutions using a set of classes taken from literature, known software libraries, and independent testing benchmarks. The comparison considers statement and branch coverage, size of generated tests, and generation time. On considered classes, TestFul generates better tests than other search-based solutions, and achieves higher structural coverages with tests small enough to be usable.","PeriodicalId":192678,"journal":{"name":"2010 Third International Conference on Software Testing, Verification and Validation","volume":"17 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2010-04-06","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"124382632","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Elisabeth Jöbstl, Martin Weiglhofer, B. Aichernig, F. Wotawa
Model-based testing is a well known technique that allows one to validate the correctness of software with respect to its model. If a lot of data is involved, symbolic techniques usually outperform explicit data enumeration. In this paper, we focus on a new symbolic test case generation technique. Our approach is based on symbolic execution and on satisfiability (modulo theory; SMT) solving. Our work was motivated by the complete failure of a well-known existing symbolic test case generator to produce any test cases for an industrial Session Initiation Protocol (SIP) implementation. Hence, we have replaced the BDD-based analysis of the existing tool with a combination of symbolic execution and SMT solving. Our new tool generates the test cases for SIP in seconds. However, further experiments showed that our approach is not a substitutive but a complementary approach: we present the technique and the results obtained for two protocol specifications, the first supporting our new technique, the second being witness for the classic BDD-technique.
{"title":"When BDDs Fail: Conformance Testing with Symbolic Execution and SMT Solving","authors":"Elisabeth Jöbstl, Martin Weiglhofer, B. Aichernig, F. Wotawa","doi":"10.1109/ICST.2010.48","DOIUrl":"https://doi.org/10.1109/ICST.2010.48","url":null,"abstract":"Model-based testing is a well known technique that allows one to validate the correctness of software with respect to its model. If a lot of data is involved, symbolic techniques usually outperform explicit data enumeration. In this paper, we focus on a new symbolic test case generation technique. Our approach is based on symbolic execution and on satisfiability (modulo theory; SMT) solving. Our work was motivated by the complete failure of a well-known existing symbolic test case generator to produce any test cases for an industrial Session Initiation Protocol (SIP) implementation. Hence, we have replaced the BDD-based analysis of the existing tool with a combination of symbolic execution and SMT solving. Our new tool generates the test cases for SIP in seconds. However, further experiments showed that our approach is not a substitutive but a complementary approach: we present the technique and the results obtained for two protocol specifications, the first supporting our new technique, the second being witness for the classic BDD-technique.","PeriodicalId":192678,"journal":{"name":"2010 Third International Conference on Software Testing, Verification and Validation","volume":"45 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2010-04-06","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"124407448","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Achim D. Brucker, Lukas Brügger, P. Kearney, B. Wolff
We present an optimization technique for model-based generation of test cases for firewalls. Starting from a formal model for firewall policies in higher-order logic, we derive a collection of semantics-preserving policy transformation rules and an algorithm that optimizes the specification with respect of the number of test cases required for path coverage. The correctness of the rules and the algorithm is established by formal proofs in Isabelle/HOL. Finally, we use the normalized policies to generate test cases with the domain-specific firewall testing tool HOL-TestGen/FW. The resulting procedure is characterized by a gain in efficiency of two orders of magnitude. It can handle configurations with hundreds of rules such as frequently occur in practice. Our approach can be seen as an instance of a methodology to tame inherent state-space explosions in test case generation for security policies.
{"title":"Verified Firewall Policy Transformations for Test Case Generation","authors":"Achim D. Brucker, Lukas Brügger, P. Kearney, B. Wolff","doi":"10.1109/ICST.2010.50","DOIUrl":"https://doi.org/10.1109/ICST.2010.50","url":null,"abstract":"We present an optimization technique for model-based generation of test cases for firewalls. Starting from a formal model for firewall policies in higher-order logic, we derive a collection of semantics-preserving policy transformation rules and an algorithm that optimizes the specification with respect of the number of test cases required for path coverage. The correctness of the rules and the algorithm is established by formal proofs in Isabelle/HOL. Finally, we use the normalized policies to generate test cases with the domain-specific firewall testing tool HOL-TestGen/FW. The resulting procedure is characterized by a gain in efficiency of two orders of magnitude. It can handle configurations with hundreds of rules such as frequently occur in practice. Our approach can be seen as an instance of a methodology to tame inherent state-space explosions in test case generation for security policies.","PeriodicalId":192678,"journal":{"name":"2010 Third International Conference on Software Testing, Verification and Validation","volume":"53 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2010-04-06","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"124613594","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Markov chains with Labelled Transitions can be used to generate test cases in a model-based approach. These test cases are generated by random walks on the model according to probabilities associated with transitions. When these probabilities correspond to a usage profile, reliability may be estimated. However, in early stages of development, such probabilities are not easy to determine, thus default profiles must be considered. In such a case it may be interesting to target some coverage criteria rather to use classical uniform probability generation approach. In this paper we enrich an existing industrial tool based on usage profile with 3 possibilities to create default profiles that improve transition coverage. We report experiments that compare the improvement of the coverage rates by our approaches with respect to uniform probabilities on transitions from a given state, which is the current default profile.
{"title":"Generating Transition Probabilities for Automatic Model-Based Test Generation","authors":"Abderrahmane Feliachi, H. L. Guen","doi":"10.1109/ICST.2010.26","DOIUrl":"https://doi.org/10.1109/ICST.2010.26","url":null,"abstract":"Markov chains with Labelled Transitions can be used to generate test cases in a model-based approach. These test cases are generated by random walks on the model according to probabilities associated with transitions. When these probabilities correspond to a usage profile, reliability may be estimated. However, in early stages of development, such probabilities are not easy to determine, thus default profiles must be considered. In such a case it may be interesting to target some coverage criteria rather to use classical uniform probability generation approach. In this paper we enrich an existing industrial tool based on usage profile with 3 possibilities to create default profiles that improve transition coverage. We report experiments that compare the improvement of the coverage rates by our approaches with respect to uniform probabilities on transitions from a given state, which is the current default profile.","PeriodicalId":192678,"journal":{"name":"2010 Third International Conference on Software Testing, Verification and Validation","volume":"6 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2010-04-06","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"127453788","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
C. Bertolini, A. Mota, E. Aranha, Cristiano Ferraz
Industry uses different testing techniques for test case generation and execution. But in general no systematic evaluation is performed to identify which technique is better (for instance, to find bugs faster). This paper presents a statistical assessment of two GUI testing techniques, BxT and DH, which are used on Motorola phone applications. These techniques test applications by pressing certain phone keys, from certain screens and during some amount of time. We consider three exploration parameters for each technique in our design and analysis of experiments: Driven determines whether a test case always starts from a single initial state (screen) or set of initial states; KeyProb associates an occurrence probability for SizeTC refers to the number of steps a test can have (a fourth parameter is the Technique itself). As conclusions, we show that BxT is better than DH and the SizeTC and the Technique parameters and the combination Driven*SizeTC have significant effects on the time to find a bug.
{"title":"GUI Testing Techniques Evaluation by Designed Experiments","authors":"C. Bertolini, A. Mota, E. Aranha, Cristiano Ferraz","doi":"10.1109/ICST.2010.41","DOIUrl":"https://doi.org/10.1109/ICST.2010.41","url":null,"abstract":"Industry uses different testing techniques for test case generation and execution. But in general no systematic evaluation is performed to identify which technique is better (for instance, to find bugs faster). This paper presents a statistical assessment of two GUI testing techniques, BxT and DH, which are used on Motorola phone applications. These techniques test applications by pressing certain phone keys, from certain screens and during some amount of time. We consider three exploration parameters for each technique in our design and analysis of experiments: Driven determines whether a test case always starts from a single initial state (screen) or set of initial states; KeyProb associates an occurrence probability for SizeTC refers to the number of steps a test can have (a fourth parameter is the Technique itself). As conclusions, we show that BxT is better than DH and the SizeTC and the Technique parameters and the combination Driven*SizeTC have significant effects on the time to find a bug.","PeriodicalId":192678,"journal":{"name":"2010 Third International Conference on Software Testing, Verification and Validation","volume":"35 2 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2010-04-06","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"116607778","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
京公网安备 11010802042870号
京ICP备2023020795号-1
扫码关注我们
求助内容：
应助结果提醒方式：