V. Garousi, W. Afzal, Adem Çaglar, I. Isik, Berker Baydan, Seçkin Çaylak, Ahmet Zeki Boyraz, Burak Yolaçan, Kadir Herkiloglu
Visual GUI testing (VGT) is a tool-driven technique, which uses image recognition for interaction and assertion of the behaviour of system under test. Motivated by a real industrial need, in the context of a large Turkish software and systems company providing solutions in the areas of defense and IT sector, we systematically planned and applied a VGT project in this industrial context. The goal of the initial phase of the project was to empirically evaluate two well-known VGT tools (Sikuli and JAutomate) to help the company select the best tool for a given testing project. Our results show that both two tools suffer from similar test ‘Replay’ problems such as the inability to find smaller-sized images. The repeatability of test executions was better for JAutomate in case of one of the two software under test (SUT) while it was comparable for the other. In terms of test development effort, for both tools, there were high correlations with number of steps in test suites, however the effort is reduced if test code is reused. The study has already provided benefits to the test engineers and managers in the company by increasing the know-how in the company w.r.t. VGT, and by identifying the challenges and their workarounds in using the tools. The industrial case study in this paper intends to add to the body of evidence in VGT and help other researchers and practitioners.
{"title":"Comparing automated visual GUI testing tools: an industrial case study","authors":"V. Garousi, W. Afzal, Adem Çaglar, I. Isik, Berker Baydan, Seçkin Çaylak, Ahmet Zeki Boyraz, Burak Yolaçan, Kadir Herkiloglu","doi":"10.1145/3121245.3121250","DOIUrl":"https://doi.org/10.1145/3121245.3121250","url":null,"abstract":"Visual GUI testing (VGT) is a tool-driven technique, which uses image recognition for interaction and assertion of the behaviour of system under test. Motivated by a real industrial need, in the context of a large Turkish software and systems company providing solutions in the areas of defense and IT sector, we systematically planned and applied a VGT project in this industrial context. The goal of the initial phase of the project was to empirically evaluate two well-known VGT tools (Sikuli and JAutomate) to help the company select the best tool for a given testing project. Our results show that both two tools suffer from similar test ‘Replay’ problems such as the inability to find smaller-sized images. The repeatability of test executions was better for JAutomate in case of one of the two software under test (SUT) while it was comparable for the other. In terms of test development effort, for both tools, there were high correlations with number of steps in test suites, however the effort is reduced if test code is reused. The study has already provided benefits to the test engineers and managers in the company by increasing the know-how in the company w.r.t. VGT, and by identifying the challenges and their workarounds in using the tools. The industrial case study in this paper intends to add to the body of evidence in VGT and help other researchers and practitioners.","PeriodicalId":107820,"journal":{"name":"Proceedings of the 8th ACM SIGSOFT International Workshop on Automated Software Testing","volume":"1 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2017-09-04","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"129143146","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Kenyo Abadio Crosara Faria, E. N. D. A. Freitas, A. Vincenzi
Collaborative Economy (CE) promotes significant changes in several sectors around the world, e.g. the famous companies Uber, Airbnb, and Turo. The general idea behind CE is the establishment of a win-win partnership between two agents. One agent has a potential need for a resource of high cost for acquisition or location, and the other agent has the resource frequently idle. Software quality verification on Android ecosystem is a hard task due to the fragmentation among the devices, i.e. the large number of devices configurations. In this scenario, compatibility testing demands the acquisition or location of several different devices of high cost and volatility due to technology evolution. On the other hand, there are several devices around the world with a high rate of idle time and which could be used for testing, generating an extra budget for their owners. In this sense, this paper defends the principles of CE for supporting the testing activity of Android applications. We implemented a platform to use and evaluate the practical usefulness and applicability of CE principles in Android software testing context. The platform makes it possible to run system testing on several devices, geographically distributed, simultaneously. The general idea is to record system testing by using an extension of Expresso, a Google proposed framework for user interface (UI) testing, and execute the test cases on idle devices previous registered on the platform, according to the test requirements. We carried out some exploratory studies which evidentiate the potential of the proposed platform, its benefits, and its impact not only on the market but also on the way we can run efficiently testing on Android ecosystem.
{"title":"Collaborative economy for testing cost reduction on Android ecosystem","authors":"Kenyo Abadio Crosara Faria, E. N. D. A. Freitas, A. Vincenzi","doi":"10.1145/3121245.3121248","DOIUrl":"https://doi.org/10.1145/3121245.3121248","url":null,"abstract":"Collaborative Economy (CE) promotes significant changes in several sectors around the world, e.g. the famous companies Uber, Airbnb, and Turo. The general idea behind CE is the establishment of a win-win partnership between two agents. One agent has a potential need for a resource of high cost for acquisition or location, and the other agent has the resource frequently idle. Software quality verification on Android ecosystem is a hard task due to the fragmentation among the devices, i.e. the large number of devices configurations. In this scenario, compatibility testing demands the acquisition or location of several different devices of high cost and volatility due to technology evolution. On the other hand, there are several devices around the world with a high rate of idle time and which could be used for testing, generating an extra budget for their owners. In this sense, this paper defends the principles of CE for supporting the testing activity of Android applications. We implemented a platform to use and evaluate the practical usefulness and applicability of CE principles in Android software testing context. The platform makes it possible to run system testing on several devices, geographically distributed, simultaneously. The general idea is to record system testing by using an extension of Expresso, a Google proposed framework for user interface (UI) testing, and execute the test cases on idle devices previous registered on the platform, according to the test requirements. We carried out some exploratory studies which evidentiate the potential of the proposed platform, its benefits, and its impact not only on the market but also on the way we can run efficiently testing on Android ecosystem.","PeriodicalId":107820,"journal":{"name":"Proceedings of the 8th ACM SIGSOFT International Workshop on Automated Software Testing","volume":"107 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2017-09-04","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"114350913","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Many software projects maintain automated GUI tests that are repeatedly executed for regression testing. Every test run executes exactly the same fixed sequence of steps confirming that the currently tested version shows precisely the same behavior as the last version. The confirmatory approach implemented by these tests limits their ability to find new defects. We therefore propose to combine existing automated regression tests with random test generation. Random test generation creates a rich variety of test steps that interact with the system under test in new, unexpected ways. Enhancing existing test cases with random test steps allows revealing new, hidden defects with little extra effort. In this paper we describe our implementation of a hybrid approach that enhances existing GUI test cases with additional, randomly generated interactions. We conducted an experiment using a mature, widely-used open source application. On average the added random interactions increased the number of visited application windows per test by 23.6% and code coverage by 12.9%. Running the enhanced tests revealed three new defects.
{"title":"Hybrid monkey testing: enhancing automated GUI tests with random test generation","authors":"Thomas Wetzlmaier, R. Ramler","doi":"10.1145/3121245.3121247","DOIUrl":"https://doi.org/10.1145/3121245.3121247","url":null,"abstract":"Many software projects maintain automated GUI tests that are repeatedly executed for regression testing. Every test run executes exactly the same fixed sequence of steps confirming that the currently tested version shows precisely the same behavior as the last version. The confirmatory approach implemented by these tests limits their ability to find new defects. We therefore propose to combine existing automated regression tests with random test generation. Random test generation creates a rich variety of test steps that interact with the system under test in new, unexpected ways. Enhancing existing test cases with random test steps allows revealing new, hidden defects with little extra effort. In this paper we describe our implementation of a hybrid approach that enhances existing GUI test cases with additional, randomly generated interactions. We conducted an experiment using a mature, widely-used open source application. On average the added random interactions increased the number of visited application windows per test by 23.6% and code coverage by 12.9%. Running the enhanced tests revealed three new defects.","PeriodicalId":107820,"journal":{"name":"Proceedings of the 8th ACM SIGSOFT International Workshop on Automated Software Testing","volume":"58 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2017-09-04","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"127905934","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Many academic studies in the field of software testing rely on mutation testing to use as their comparison criteria. However, recent studies have shown that redundant mutants have a significant effect on the accuracy of their results. One solution to this problem is to use mutant subsumption to detect redundant mutants. Therefore, in order to facilitate research in this field, a mutation testing tool that is capable of detecting redundant mutants is needed. In this paper, we describe how we improved our tool, LittleDarwin, to fulfill this requirement.
{"title":"Dynamic mutant subsumption analysis using LittleDarwin","authors":"Ali Parsai, S. Demeyer","doi":"10.1145/3121245.3121249","DOIUrl":"https://doi.org/10.1145/3121245.3121249","url":null,"abstract":"Many academic studies in the field of software testing rely on mutation testing to use as their comparison criteria. However, recent studies have shown that redundant mutants have a significant effect on the accuracy of their results. One solution to this problem is to use mutant subsumption to detect redundant mutants. Therefore, in order to facilitate research in this field, a mutation testing tool that is capable of detecting redundant mutants is needed. In this paper, we describe how we improved our tool, LittleDarwin, to fulfill this requirement.","PeriodicalId":107820,"journal":{"name":"Proceedings of the 8th ACM SIGSOFT International Workshop on Automated Software Testing","volume":"16 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2017-09-04","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"127753379","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
P. Krishnan, Jerome Loh, Rebecca O'Donoghue, L. Meinicke
In this position paper we describe how mutation testing can be used to evaluate the quality of test suites from a security viewpoint. Our focus is on measuring the quality of the test suite associated with the Java Development Kit (JDK) because it provides the core security properties for all applications. We describe the challenges associated with identifying security-specific mutation operators that are specific to the Java model and ensuring that our solution can be automated for large code-bases like the JDK.
{"title":"Evaluating quality of security testing of the JDK","authors":"P. Krishnan, Jerome Loh, Rebecca O'Donoghue, L. Meinicke","doi":"10.1145/3121245.3121246","DOIUrl":"https://doi.org/10.1145/3121245.3121246","url":null,"abstract":"In this position paper we describe how mutation testing can be used to evaluate the quality of test suites from a security viewpoint. Our focus is on measuring the quality of the test suite associated with the Java Development Kit (JDK) because it provides the core security properties for all applications. We describe the challenges associated with identifying security-specific mutation operators that are specific to the Java model and ensuring that our solution can be automated for large code-bases like the JDK.","PeriodicalId":107820,"journal":{"name":"Proceedings of the 8th ACM SIGSOFT International Workshop on Automated Software Testing","volume":"106 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2017-09-04","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"122630484","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Proceedings of the 8th ACM SIGSOFT International Workshop on Automated Software Testing","authors":"T. Vos, Sigrid Eldh, Wishnu Prasetya","doi":"10.1145/3121245","DOIUrl":"https://doi.org/10.1145/3121245","url":null,"abstract":"","PeriodicalId":107820,"journal":{"name":"Proceedings of the 8th ACM SIGSOFT International Workshop on Automated Software Testing","volume":"10 2 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2017-09-04","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"116654620","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
京公网安备 11010802042870号
京ICP备2023020795号-1
扫码关注我们
求助内容：
应助结果提醒方式：