Fault localization techniques have been applied to database and data-centric applications that use SQL or SQL-based languages. However, existing techniques can only identify the SQL statements that have faults, but not determine the precise location of the faults within SQL statements. Since SQL statements can be rather complex, programmers are still left with a difficult repair chore. We propose a novel fault localization method to localize multiple types of faults in SQL predicates, that is based on row-based dynamic slicing and delta debugging. Our method was implemented in a tool called ALTAR, and experiments were performed on two publicly available databases. Our method can be compared with existing fault localization techniques when these are applied to "drill-down" in SQL statements. The results showed that ALTAR can discover more types of faults. Moreover, for the type of faults discovered by current methods, ALTAR is more precise.
{"title":"Localizing Faults in SQL Predicates","authors":"Yun Guo, Amihai Motro, Nan Li","doi":"10.1109/ICST.2017.8","DOIUrl":"https://doi.org/10.1109/ICST.2017.8","url":null,"abstract":"Fault localization techniques have been applied to database and data-centric applications that use SQL or SQL-based languages. However, existing techniques can only identify the SQL statements that have faults, but not determine the precise location of the faults within SQL statements. Since SQL statements can be rather complex, programmers are still left with a difficult repair chore. We propose a novel fault localization method to localize multiple types of faults in SQL predicates, that is based on row-based dynamic slicing and delta debugging. Our method was implemented in a tool called ALTAR, and experiments were performed on two publicly available databases. Our method can be compared with existing fault localization techniques when these are applied to \"drill-down\" in SQL statements. The results showed that ALTAR can discover more types of faults. Moreover, for the type of faults discovered by current methods, ALTAR is more precise.","PeriodicalId":112258,"journal":{"name":"2017 IEEE International Conference on Software Testing, Verification and Validation (ICST)","volume":"118 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2017-03-13","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"128179405","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Unit testing is an essential practice in Extreme Programming (XP) and Test-driven Development (TDD) and used in many software lifecycle models. Additionally, a lot of literature deals with this topic. Therefore, it can be expected that it is widely used among developers. Despite its importance, there is no empirical study which investigates, whether unit tests are used by developers in real life projects at all. This paper presents such a study, where we collected and analyzed data from over 70K revisions of 10 different Python projects. Based on two different definitions of unit testing, we calculated the actual number of unit tests and compared it with the expected number (as inferred from the intentions of the developers), had a look at the mocking behavior of developers, and at the evolution of the number of unit tests. Our main findings show, (i) that developers believe that they are developing more unit tests than they actually do, (ii) most projects have a very small amount of unit tests, (iii) developers make use of mocks, but these do not have a significant influence on the number of unit tests, (iv) four different patterns for the evolution of the number of unit tests could be detected, and (v) the used unit test definition has an influence on the results.
{"title":"Are There Any Unit Tests? An Empirical Study on Unit Testing in Open Source Python Projects","authors":"Fabian Trautsch, J. Grabowski","doi":"10.1109/ICST.2017.26","DOIUrl":"https://doi.org/10.1109/ICST.2017.26","url":null,"abstract":"Unit testing is an essential practice in Extreme Programming (XP) and Test-driven Development (TDD) and used in many software lifecycle models. Additionally, a lot of literature deals with this topic. Therefore, it can be expected that it is widely used among developers. Despite its importance, there is no empirical study which investigates, whether unit tests are used by developers in real life projects at all. This paper presents such a study, where we collected and analyzed data from over 70K revisions of 10 different Python projects. Based on two different definitions of unit testing, we calculated the actual number of unit tests and compared it with the expected number (as inferred from the intentions of the developers), had a look at the mocking behavior of developers, and at the evolution of the number of unit tests. Our main findings show, (i) that developers believe that they are developing more unit tests than they actually do, (ii) most projects have a very small amount of unit tests, (iii) developers make use of mocks, but these do not have a significant influence on the number of unit tests, (iv) four different patterns for the evolution of the number of unit tests could be detected, and (v) the used unit test definition has an influence on the results.","PeriodicalId":112258,"journal":{"name":"2017 IEEE International Conference on Software Testing, Verification and Validation (ICST)","volume":"37 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2017-03-13","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"134250938","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
NuSMV is a well-known tool for system verification that permits to verify both CTL and LTL properties. Although the tool is very powerful, it offers a minimal support for the editing and validation (e.g., by simulation) of models and of requirements specified as temporal properties. In this paper, we propose NuSeen, a framework that assists a designer during the modeling and V&V activities when using NuSMV. In addition to an editor furnished with syntax highlighting, autocompletion, and outline, NuSeen also provides some tools for visualizing the variable dependencies, and graphically visualizing the counterexamples. It helps the designer in validating the model by checking certain qualities like minimality and completeness. Moreover, the framework also provides facilities for model-based testing by means of a test suite generator that is able to generate tests achieving value and decision coverage for NuSMV models.
{"title":"NuSeen: A Tool Framework for the NuSMV Model Checker","authors":"Paolo Arcaini, A. Gargantini, E. Riccobene","doi":"10.1109/ICST.2017.54","DOIUrl":"https://doi.org/10.1109/ICST.2017.54","url":null,"abstract":"NuSMV is a well-known tool for system verification that permits to verify both CTL and LTL properties. Although the tool is very powerful, it offers a minimal support for the editing and validation (e.g., by simulation) of models and of requirements specified as temporal properties. In this paper, we propose NuSeen, a framework that assists a designer during the modeling and V&V activities when using NuSMV. In addition to an editor furnished with syntax highlighting, autocompletion, and outline, NuSeen also provides some tools for visualizing the variable dependencies, and graphically visualizing the counterexamples. It helps the designer in validating the model by checking certain qualities like minimality and completeness. Moreover, the framework also provides facilities for model-based testing by means of a test suite generator that is able to generate tests achieving value and decision coverage for NuSMV models.","PeriodicalId":112258,"journal":{"name":"2017 IEEE International Conference on Software Testing, Verification and Validation (ICST)","volume":"96 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2017-03-13","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"124484124","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Xiao Li, N. Chang, Yan Wang, Haohua Huang, Yu Pei, Linzhang Wang, Xuandong Li
The importance of regression testing in assuring the integrity of a program after changes is well recognized. One major obstacle in practicing regression testing is in maintaining tests that become obsolete due to evolved program behavior or specification. For mobile apps, the problem of maintaining obsolete GUI test scripts for regression testing is even more pressing. Mobile apps rely heavily on the correct functioning of their GUIs to compete on the market and provide good user experiences. But on the one hand, GUI tests break easily when changes happen to the GUI, On the other hand, mobile app developers often need to fight for a tight feedback loop and are left with limited time for test maintenance. In this paper, we propose a novel approach, called ATOM, to automatically maintain GUI test scripts of mobile apps for regression testing. ATOM uses an event sequence model to abstract possible event sequences on a GUI and a delta ESM to abstract the changes made to the GUI. Given both models as input, ATOM automatically updates the test scripts written for a base version app to reflect the changes. In an experiment with 22 versions from 11 production Android apps, ATOM updated all the test scripts affected by the version change, the updated scripts achieve over 80% of the coverage by the original scripts on the base version app, all except one set of updated scripts preserve over 60% of the actions in the original test scripts.
{"title":"ATOM: Automatic Maintenance of GUI Test Scripts for Evolving Mobile Applications","authors":"Xiao Li, N. Chang, Yan Wang, Haohua Huang, Yu Pei, Linzhang Wang, Xuandong Li","doi":"10.1109/ICST.2017.22","DOIUrl":"https://doi.org/10.1109/ICST.2017.22","url":null,"abstract":"The importance of regression testing in assuring the integrity of a program after changes is well recognized. One major obstacle in practicing regression testing is in maintaining tests that become obsolete due to evolved program behavior or specification. For mobile apps, the problem of maintaining obsolete GUI test scripts for regression testing is even more pressing. Mobile apps rely heavily on the correct functioning of their GUIs to compete on the market and provide good user experiences. But on the one hand, GUI tests break easily when changes happen to the GUI, On the other hand, mobile app developers often need to fight for a tight feedback loop and are left with limited time for test maintenance. In this paper, we propose a novel approach, called ATOM, to automatically maintain GUI test scripts of mobile apps for regression testing. ATOM uses an event sequence model to abstract possible event sequences on a GUI and a delta ESM to abstract the changes made to the GUI. Given both models as input, ATOM automatically updates the test scripts written for a base version app to reflect the changes. In an experiment with 22 versions from 11 production Android apps, ATOM updated all the test scripts affected by the version change, the updated scripts achieve over 80% of the coverage by the original scripts on the base version app, all except one set of updated scripts preserve over 60% of the actions in the original test scripts.","PeriodicalId":112258,"journal":{"name":"2017 IEEE International Conference on Software Testing, Verification and Validation (ICST)","volume":"08 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2017-03-13","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"130661436","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Several fault predictors were proposed in the context of Spectrum-based Fault Localization approaches to rank software components in order of suspiciousness of being the root-cause of observed failures. Previous work has also shown that some of the fault predictors (near-)optimally rank software components, provided that there is one fault in the system. Despite this, further work is being spent on creating more complex, computationally expensive, model-based techniques that can handle multiple-faulted scenarios accurately. However, our hypothesis is that when software is being developed, bugs arise one-at-a-time and therefore can be considered as single-faulted scenarios. We describe an approach to mine repositories, find bug-fixes, and catalog them according to the number of faults they fix, to assess the prevalence of single-fault fixes. Our empirical study using 279 open-source projects reveals that there is a prevalence of single-fault fixes, with over 82% of all fixes only eliminating one bug from the system, enabling the use of simpler, (near-)optimal, fault predictors. Moreover, we draw on the practical implications of our findings to influence and set direction for future research.
{"title":"Prevalence of Single-Fault Fixes and Its Impact on Fault Localization","authors":"Alexandre Perez, Rui Abreu, Marcelo d’Amorim","doi":"10.1109/ICST.2017.9","DOIUrl":"https://doi.org/10.1109/ICST.2017.9","url":null,"abstract":"Several fault predictors were proposed in the context of Spectrum-based Fault Localization approaches to rank software components in order of suspiciousness of being the root-cause of observed failures. Previous work has also shown that some of the fault predictors (near-)optimally rank software components, provided that there is one fault in the system. Despite this, further work is being spent on creating more complex, computationally expensive, model-based techniques that can handle multiple-faulted scenarios accurately. However, our hypothesis is that when software is being developed, bugs arise one-at-a-time and therefore can be considered as single-faulted scenarios. We describe an approach to mine repositories, find bug-fixes, and catalog them according to the number of faults they fix, to assess the prevalence of single-fault fixes. Our empirical study using 279 open-source projects reveals that there is a prevalence of single-fault fixes, with over 82% of all fixes only eliminating one bug from the system, enabling the use of simpler, (near-)optimal, fault predictors. Moreover, we draw on the practical implications of our findings to influence and set direction for future research.","PeriodicalId":112258,"journal":{"name":"2017 IEEE International Conference on Software Testing, Verification and Validation (ICST)","volume":"37 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2017-03-13","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"123659515","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
In the context of use-case centric development and requirements-driven testing, this paper addresses the problem of automatically deriving system test cases to verify timing requirements. Inspired by engineering practice in an automotive software development context, we rely on an analyzable form of use case specifications and augment such functional descriptions with timed automata, capturing timing requirements, following a methodology aiming at minimizing modeling overhead. We automate the generation of executable test cases using a test strategy based on maximizing test suite diversity and building over the UPPAAL model checker. Initial empirical results based on an industrial case study provide evidence of the effectiveness of the approach.
{"title":"System Testing of Timing Requirements Based on Use Cases and Timed Automata","authors":"Chunhui Wang, F. Pastore, L. Briand","doi":"10.1109/ICST.2017.34","DOIUrl":"https://doi.org/10.1109/ICST.2017.34","url":null,"abstract":"In the context of use-case centric development and requirements-driven testing, this paper addresses the problem of automatically deriving system test cases to verify timing requirements. Inspired by engineering practice in an automotive software development context, we rely on an analyzable form of use case specifications and augment such functional descriptions with timed automata, capturing timing requirements, following a methodology aiming at minimizing modeling overhead. We automate the generation of executable test cases using a test strategy based on maximizing test suite diversity and building over the UPPAAL model checker. Initial empirical results based on an industrial case study provide evidence of the effectiveness of the approach.","PeriodicalId":112258,"journal":{"name":"2017 IEEE International Conference on Software Testing, Verification and Validation (ICST)","volume":"103 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2017-03-13","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"125874317","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Xiaodong Zhang, Z. Yang, Q. Zheng, Pei Liu, Jialiang Chang, Yu Hao, Ting Liu
With the advent of multicore processors, there is a trend towards multithreading to take advantage of parallel computing resources. Due to greatly increased complexity, programmers need effective testing methodology that can thoroughly test multithreaded programs. There has been significant progress based on symbolic execution that attempts to exhaustively explore all the intra-thread paths and inter-thread interleavings. However, such testing approach faces two insuperable challenges. Firstly, exploring an astronomically large number of paths and interleavings limits its scalability. Secondly, a path itself does not directly help programmers understand program behavior. In this paper, we propose an alternate testing methodology that focuses on definition-use data flow instead of paths/interleavings. Such approach not only leads to orders of magnitude reduction in testing complexity, but also gives programmers direct help on examining the shared variable usage in a multithreaded program.
{"title":"Automated Testing of Definition-Use Data Flow for Multithreaded Programs","authors":"Xiaodong Zhang, Z. Yang, Q. Zheng, Pei Liu, Jialiang Chang, Yu Hao, Ting Liu","doi":"10.1109/ICST.2017.23","DOIUrl":"https://doi.org/10.1109/ICST.2017.23","url":null,"abstract":"With the advent of multicore processors, there is a trend towards multithreading to take advantage of parallel computing resources. Due to greatly increased complexity, programmers need effective testing methodology that can thoroughly test multithreaded programs. There has been significant progress based on symbolic execution that attempts to exhaustively explore all the intra-thread paths and inter-thread interleavings. However, such testing approach faces two insuperable challenges. Firstly, exploring an astronomically large number of paths and interleavings limits its scalability. Secondly, a path itself does not directly help programmers understand program behavior. In this paper, we propose an alternate testing methodology that focuses on definition-use data flow instead of paths/interleavings. Such approach not only leads to orders of magnitude reduction in testing complexity, but also gives programmers direct help on examining the shared variable usage in a multithreaded program.","PeriodicalId":112258,"journal":{"name":"2017 IEEE International Conference on Software Testing, Verification and Validation (ICST)","volume":"187 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2017-03-13","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"121096166","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
M. Fazzini, E. N. D. A. Freitas, Shauvik Roy Choudhary, A. Orso
Because mobile apps are extremely popular and often mission critical nowadays, companies invest a great deal of resources in testing the apps they provide to their customers. Testing is particularly important for Android apps, which must run on a multitude of devices and operating system versions. Unfortunately, as we confirmed in many interviews with quality assurance professionals, app testing is today a very human intensive, and therefore tedious and error prone, activity. To address this problem, and better support testing of Android apps, we propose a new technique that allows testers to easily create platform independent test scripts for an app and automatically run the generated test scripts on multiple devices and operating system versions. The technique does so without modifying the app under test or the runtime system, by (1) intercepting the interactions of the tester with the app and (2) providing the tester with an intuitive way to specify expected results that it then encode as test oracles. We implemented our technique in a tool named Barista and used the tool to evaluate the practical usefulness and applicability of our approach. Our results show that Barista (1) can faithfully encode user defined test cases as test scripts with built-in oracles that can run on multiple platforms and (2) outperforms two popular tools with similar functionality. Barista and our experimental infrastructure are publicly available.
{"title":"Barista: A Technique for Recording, Encoding, and Running Platform Independent Android Tests","authors":"M. Fazzini, E. N. D. A. Freitas, Shauvik Roy Choudhary, A. Orso","doi":"10.1109/ICST.2017.21","DOIUrl":"https://doi.org/10.1109/ICST.2017.21","url":null,"abstract":"Because mobile apps are extremely popular and often mission critical nowadays, companies invest a great deal of resources in testing the apps they provide to their customers. Testing is particularly important for Android apps, which must run on a multitude of devices and operating system versions. Unfortunately, as we confirmed in many interviews with quality assurance professionals, app testing is today a very human intensive, and therefore tedious and error prone, activity. To address this problem, and better support testing of Android apps, we propose a new technique that allows testers to easily create platform independent test scripts for an app and automatically run the generated test scripts on multiple devices and operating system versions. The technique does so without modifying the app under test or the runtime system, by (1) intercepting the interactions of the tester with the app and (2) providing the tester with an intuitive way to specify expected results that it then encode as test oracles. We implemented our technique in a tool named Barista and used the tool to evaluate the practical usefulness and applicability of our approach. Our results show that Barista (1) can faithfully encode user defined test cases as test scripts with built-in oracles that can run on multiple platforms and (2) outperforms two popular tools with similar functionality. Barista and our experimental infrastructure are publicly available.","PeriodicalId":112258,"journal":{"name":"2017 IEEE International Conference on Software Testing, Verification and Validation (ICST)","volume":"1 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2017-03-13","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"128750806","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
We consider regression testing of safety-critical systems consisting of black-box components. This scenario is common for automotive electronic systems where testing time is expensive and should be reduced without uncontrolled reduction of reliability. This requires a methodology to select test cases from a larger test suite for which a defined quality level of the resulting regression test cycle can be provided. With this in mind, we propose a method to select test cases based on a stochastic model. We are modeling test case failure probabilities as dependent random variables, therefore already observed test results have an influence on the estimation of further test case failure probabilities. Based on an information theoretical approach, we validate the mutual differential information degree between test case failure probabilities and compute a function which returns the risk for not selecting a test case, i.e., the probability of not selecting a failing test case. Depending on the mutual differential information significant reductions of testing time can be achieved while testing reliability is preserved at a quantifiable high level. We will validate theoretically our results and will show in an industrial case study the benefits of our method.
{"title":"A Selection Method for Black Box Regression Testing with a Statistically Defined Quality Level","authors":"Ibrahim Alagöz, T. Herpel, R. German","doi":"10.1109/ICST.2017.18","DOIUrl":"https://doi.org/10.1109/ICST.2017.18","url":null,"abstract":"We consider regression testing of safety-critical systems consisting of black-box components. This scenario is common for automotive electronic systems where testing time is expensive and should be reduced without uncontrolled reduction of reliability. This requires a methodology to select test cases from a larger test suite for which a defined quality level of the resulting regression test cycle can be provided. With this in mind, we propose a method to select test cases based on a stochastic model. We are modeling test case failure probabilities as dependent random variables, therefore already observed test results have an influence on the estimation of further test case failure probabilities. Based on an information theoretical approach, we validate the mutual differential information degree between test case failure probabilities and compute a function which returns the risk for not selecting a test case, i.e., the probability of not selecting a failing test case. Depending on the mutual differential information significant reductions of testing time can be achieved while testing reliability is preserved at a quantifiable high level. We will validate theoretically our results and will show in an industrial case study the benefits of our method.","PeriodicalId":112258,"journal":{"name":"2017 IEEE International Conference on Software Testing, Verification and Validation (ICST)","volume":"27 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2017-03-13","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"124022562","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
D. Marijan, Marius Liaaen, A. Gotlieb, S. Sen, Carlo Ieva
Exhaustive testing of highly configurable software developed in continuous integration is rarely feasible in practice due to the configuration space of exponential size on the one hand, and strict time constraints on the other. This entails using selective testing techniques to determine the most failure-inducing test cases, conforming to highly-constrained time budget. These challenges have been well recognized by researchers, such that many different techniques have been proposed. In practice, however, there is a lack of efficient tools able to reduce high testing effort, without compromising software quality. In this paper we propose a test suite optimization technology TITAN, which increases the time-and cost-efficiency of testing highly configurable software developed in continuous integration. The technology implements practical test prioritization and minimization techniques, and provides test traceability and visualization for improving the quality of testing. We present the TITAN tool and discuss a set of methodological and technological challenges we have faced during TITAN development. We evaluate TITAN in testing of Cisco's highly configurable software with frequent high quality releases, and demonstrate the benefit of the approach in such a complex industry domain.
{"title":"TITAN: Test Suite Optimization for Highly Configurable Software","authors":"D. Marijan, Marius Liaaen, A. Gotlieb, S. Sen, Carlo Ieva","doi":"10.1109/ICST.2017.60","DOIUrl":"https://doi.org/10.1109/ICST.2017.60","url":null,"abstract":"Exhaustive testing of highly configurable software developed in continuous integration is rarely feasible in practice due to the configuration space of exponential size on the one hand, and strict time constraints on the other. This entails using selective testing techniques to determine the most failure-inducing test cases, conforming to highly-constrained time budget. These challenges have been well recognized by researchers, such that many different techniques have been proposed. In practice, however, there is a lack of efficient tools able to reduce high testing effort, without compromising software quality. In this paper we propose a test suite optimization technology TITAN, which increases the time-and cost-efficiency of testing highly configurable software developed in continuous integration. The technology implements practical test prioritization and minimization techniques, and provides test traceability and visualization for improving the quality of testing. We present the TITAN tool and discuss a set of methodological and technological challenges we have faced during TITAN development. We evaluate TITAN in testing of Cisco's highly configurable software with frequent high quality releases, and demonstrate the benefit of the approach in such a complex industry domain.","PeriodicalId":112258,"journal":{"name":"2017 IEEE International Conference on Software Testing, Verification and Validation (ICST)","volume":"66 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2017-03-13","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"128642741","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
京公网安备 11010802042870号
京ICP备2023020795号-1
扫码关注我们
求助内容：
应助结果提醒方式：