André Reichstaller, Benedikt Eberhardinger, Hella Ponsar, Alexander Knapp, W. Reif
We study regression testing and test suite reduction for self-organizing (SO) systems. The complex environments of SO systems typically require large test suites. The physical distribution of their components and their history-dependent behavior, however, make test execution very expensive. Consequently, an efficient test suite reduction mechanism is needed. The fundamental characteristic of SO systems is their ability to reconfigure themselves. We thus in- vestigate a mutation-based approach concentrating on reconfigura- tions, more specifically the communication between the distributed components in reconfigurations. Due to distribution, we argue for an explicit consideration of higher-order mutants and find a short- cut that makes the number of test cases to execute before reduction feasible. For the reduction task, we evaluate the applicability of two existing clustering techniques, Affinity Propagation and Dissimilar- ity-based Sparse Subset Selection. It turns out that these techniques are able to drastically reduce the original test suite while retaining a good mutation score. We discuss the approach by means of a test suite for a self-organizing production cell as a running example.
{"title":"Test Suite Reduction for Self-Organizing Systems: A Mutation-Based Approach","authors":"André Reichstaller, Benedikt Eberhardinger, Hella Ponsar, Alexander Knapp, W. Reif","doi":"10.1145/3194733.3194739","DOIUrl":"https://doi.org/10.1145/3194733.3194739","url":null,"abstract":"We study regression testing and test suite reduction for self-organizing (SO) systems. The complex environments of SO systems typically require large test suites. The physical distribution of their components and their history-dependent behavior, however, make test execution very expensive. Consequently, an efficient test suite reduction mechanism is needed. The fundamental characteristic of SO systems is their ability to reconfigure themselves. We thus in- vestigate a mutation-based approach concentrating on reconfigura- tions, more specifically the communication between the distributed components in reconfigurations. Due to distribution, we argue for an explicit consideration of higher-order mutants and find a short- cut that makes the number of test cases to execute before reduction feasible. For the reduction task, we evaluate the applicability of two existing clustering techniques, Affinity Propagation and Dissimilar- ity-based Sparse Subset Selection. It turns out that these techniques are able to drastically reduce the original test suite while retaining a good mutation score. We discuss the approach by means of a test suite for a self-organizing production cell as a running example.","PeriodicalId":423703,"journal":{"name":"2018 IEEE/ACM 13th International Workshop on Automation of Software Test (AST)","volume":"78 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2018-05-28","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"125493484","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
F. D. O. Neto, Azeem Ahmad, O. Leifler, K. Sandahl, Eduard Paul Enoiu
Automated testing is an essential component of Continuous Integration (CI) and Delivery (CD), such as scheduling automated test sessions on overnight builds. That allows stakeholders to execute entire test suites and achieve exhaustive test coverage, since running all tests is often infeasible during work hours, i.e., in parallel to development activities. On the other hand, developers also need test feedback from CI servers when pushing changes, even if not all test cases are executed. In this paper we evaluate similarity-based test case selection (SBTCS) on integration-level tests executed on continuous integration pipelines of two companies. We select test cases that maximise diversity of test coverage and reduce feedback time to developers. Our results confirm existing evidence that SBTCS is a strong candidate for test optimisation, by reducing feedback time (up to 92% faster in our case studies) while achieving full test coverage using only information from test artefacts themselves.
{"title":"Improving Continuous Integration with Similarity-Based Test Case Selection","authors":"F. D. O. Neto, Azeem Ahmad, O. Leifler, K. Sandahl, Eduard Paul Enoiu","doi":"10.1145/3194733.3194744","DOIUrl":"https://doi.org/10.1145/3194733.3194744","url":null,"abstract":"Automated testing is an essential component of Continuous Integration (CI) and Delivery (CD), such as scheduling automated test sessions on overnight builds. That allows stakeholders to execute entire test suites and achieve exhaustive test coverage, since running all tests is often infeasible during work hours, i.e., in parallel to development activities. On the other hand, developers also need test feedback from CI servers when pushing changes, even if not all test cases are executed. In this paper we evaluate similarity-based test case selection (SBTCS) on integration-level tests executed on continuous integration pipelines of two companies. We select test cases that maximise diversity of test coverage and reduce feedback time to developers. Our results confirm existing evidence that SBTCS is a strong candidate for test optimisation, by reducing feedback time (up to 92% faster in our case studies) while achieving full test coverage using only information from test artefacts themselves.","PeriodicalId":423703,"journal":{"name":"2018 IEEE/ACM 13th International Workshop on Automation of Software Test (AST)","volume":"44 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2018-05-28","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"129507529","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Web applications are deployed on machines around the globe and offer almost universal accessibility. The systems ensure functional interconnectivity between different components on a 24/7 basis. One of the most important requirements represents data confidentiality and secure authentication. However, implementation flaws and unfulfilled requirements can result in security leaks that can be eventually exploited by a malicious user. Here different testing methods are applied in order to detect software defects and prevent unauthorized access in advance. Automated planning and scheduling provides the possibility to specify a specific problem and to generate plans, which in turn guide the execution of a program. In this paper, a planning-based approach is introduced for modeling and testing of web applications. The specification offers a high degree of extendibility and configurability but overcomes the limits of traditional graphical representations as well. In this way, new testing possibilities emerge that eventually lead to better vulnerability detection, thereby ensuring more secure services.
{"title":"Planning-Based Security Testing of Web Applications","authors":"Josip Bozic, F. Wotawa","doi":"10.1145/3194733.3194738","DOIUrl":"https://doi.org/10.1145/3194733.3194738","url":null,"abstract":"Web applications are deployed on machines around the globe and offer almost universal accessibility. The systems ensure functional interconnectivity between different components on a 24/7 basis. One of the most important requirements represents data confidentiality and secure authentication. However, implementation flaws and unfulfilled requirements can result in security leaks that can be eventually exploited by a malicious user. Here different testing methods are applied in order to detect software defects and prevent unauthorized access in advance. Automated planning and scheduling provides the possibility to specify a specific problem and to generate plans, which in turn guide the execution of a program. In this paper, a planning-based approach is introduced for modeling and testing of web applications. The specification offers a high degree of extendibility and configurability but overcomes the limits of traditional graphical representations as well. In this way, new testing possibilities emerge that eventually lead to better vulnerability detection, thereby ensuring more secure services.","PeriodicalId":423703,"journal":{"name":"2018 IEEE/ACM 13th International Workshop on Automation of Software Test (AST)","volume":"3 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2018-05-28","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"123909410","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Jihyun Park, Chang-Seo Park, Byoungju Choi, Gihun Chang
Memory fault detection has been continuously studied and various detection methods exist. However, there are still remains many memory defects that are difficult to debug. Memory corruption is one of those defects that often cause a system crash. However, there are many cases where the location of the crash is different from the actual location causing the actual memory corruption. These defects are difficult to solve by existing methods. In this paper, we propose a method to detect real time memory defects by using static global variables derived from execution binary file and dynamic memory usage obtained by tracing memory related functions. We implemented the proposed method as a tool and applied it to the application running on the IoTivity platform. Our tool detects defects very accurately with low overhead even for those whose detected location and the location of its cause are different.
{"title":"Memory Corruption Detecting Method Using Static Variables and Dynamic Memory Usage","authors":"Jihyun Park, Chang-Seo Park, Byoungju Choi, Gihun Chang","doi":"10.1145/3194733.3194741","DOIUrl":"https://doi.org/10.1145/3194733.3194741","url":null,"abstract":"Memory fault detection has been continuously studied and various detection methods exist. However, there are still remains many memory defects that are difficult to debug. Memory corruption is one of those defects that often cause a system crash. However, there are many cases where the location of the crash is different from the actual location causing the actual memory corruption. These defects are difficult to solve by existing methods. In this paper, we propose a method to detect real time memory defects by using static global variables derived from execution binary file and dynamic memory usage obtained by tracing memory related functions. We implemented the proposed method as a tool and applied it to the application running on the IoTivity platform. Our tool detects defects very accurately with low overhead even for those whose detected location and the location of its cause are different.","PeriodicalId":423703,"journal":{"name":"2018 IEEE/ACM 13th International Workshop on Automation of Software Test (AST)","volume":"2 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2018-05-28","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"131816940","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Random testing of Android apps is attractive due to ease-of-use and scalability, but its effectiveness could be questioned. Prior studies have shown that Monkey – a simple approach and tool for random testing of Android apps – is surprisingly effective, "beating" much more sophisticated tools by achieving higher coverage. We study how Monkey's parameters affect code coverage (at class, method, block, and line levels) and set out to answer several research questions centered around improving the effectiveness of Monkey-based random testing in Android, and how it compares with manual exploration. First, we show that random stress testing via Monkey is extremely efficient (85 seconds on average) and effective at crashing apps, including 15 widely-used apps that have millions (or even billions) of installs. Second, we vary Monkey's event distribution to change app behavior and measured the resulting coverage. We found that, except for isolated cases, altering Monkey's default event distribution is unlikely to lead to higher coverage. Third, we manually explore 62 apps and compare the resulting coverages; we found that coverage achieved via manual exploration is just 2-3% higher than that achieved via Monkey exploration. Finally, our analysis shows that coarse-grained coverage is highly indicative of fine-grained coverage, hence coarse-grained coverage (which imposes low collection overhead) hits a performance vs accuracy sweet spot.
{"title":"On the Effectiveness of Random Testing for Android: Or How I Learned to Stop Worrying and Love the Monkey","authors":"Priyam Patel, Gokul Srinivasan, Sydur Rahaman, Iulian Neamtiu","doi":"10.1145/3194733.3194742","DOIUrl":"https://doi.org/10.1145/3194733.3194742","url":null,"abstract":"Random testing of Android apps is attractive due to ease-of-use and scalability, but its effectiveness could be questioned. Prior studies have shown that Monkey – a simple approach and tool for random testing of Android apps – is surprisingly effective, \"beating\" much more sophisticated tools by achieving higher coverage. We study how Monkey's parameters affect code coverage (at class, method, block, and line levels) and set out to answer several research questions centered around improving the effectiveness of Monkey-based random testing in Android, and how it compares with manual exploration. First, we show that random stress testing via Monkey is extremely efficient (85 seconds on average) and effective at crashing apps, including 15 widely-used apps that have millions (or even billions) of installs. Second, we vary Monkey's event distribution to change app behavior and measured the resulting coverage. We found that, except for isolated cases, altering Monkey's default event distribution is unlikely to lead to higher coverage. Third, we manually explore 62 apps and compare the resulting coverages; we found that coverage achieved via manual exploration is just 2-3% higher than that achieved via Monkey exploration. Finally, our analysis shows that coarse-grained coverage is highly indicative of fine-grained coverage, hence coarse-grained coverage (which imposes low collection overhead) hits a performance vs accuracy sweet spot.","PeriodicalId":423703,"journal":{"name":"2018 IEEE/ACM 13th International Workshop on Automation of Software Test (AST)","volume":"127 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2018-05-28","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"124222102","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Today's enterprise software systems are much complicated than the past. Increasing number of dependent applications, heterogeneous technologies and wide usage of Service Oriented Architectures (SOA), where numerous services communicate with each other, makes testing of such systems challenging. For testing these software systems, the concept of service virtualization is gaining popularity. Service virtualization is an automated technique to mimic the behavior of a given real service. Services can be classified as stateless or stateful services. Many services are stateful in nature. Although there are works in the literature for virtualization of stateless services, no such solution exists for stateful services. To the best of our knowledge, this is the first work for stateful service virtualization. We employ classification based and sequence-to-sequence based machine learning algorithms in developing our solutions. We demonstrate the validity of our approach on two data sets collected from real life services and obtain promising results.
{"title":"Testing Service Oriented Architectures Using Stateful Service Virtualization via Machine Learning","authors":"Hasan Ferit Eniser, A. Sen","doi":"10.1145/3194733.3194737","DOIUrl":"https://doi.org/10.1145/3194733.3194737","url":null,"abstract":"Today's enterprise software systems are much complicated than the past. Increasing number of dependent applications, heterogeneous technologies and wide usage of Service Oriented Architectures (SOA), where numerous services communicate with each other, makes testing of such systems challenging. For testing these software systems, the concept of service virtualization is gaining popularity. Service virtualization is an automated technique to mimic the behavior of a given real service. Services can be classified as stateless or stateful services. Many services are stateful in nature. Although there are works in the literature for virtualization of stateless services, no such solution exists for stateful services. To the best of our knowledge, this is the first work for stateful service virtualization. We employ classification based and sequence-to-sequence based machine learning algorithms in developing our solutions. We demonstrate the validity of our approach on two data sets collected from real life services and obtain promising results.","PeriodicalId":423703,"journal":{"name":"2018 IEEE/ACM 13th International Workshop on Automation of Software Test (AST)","volume":"1 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2018-05-28","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"128566793","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
A. Bertolino, Said Daoudagh, F. Lonetti, E. Marchetti
In the context of XACML-based access control systems, an intensive testing activity is among the most adopted means to assure that sensible information or resources are correctly accessed. Unfortunately, it requires a huge effort for manual inspection of results: thus automated verdict derivation is a key aspect for improving the cost-effectiveness of testing. To this purpose, we introduce XACMET, a novel approach for automated model-based oracle definition. XACMET defines a typed graph, called the XAC-Graph, that models the XACML policy evaluation. The expected verdict of a specific request execution can thus be automatically derived by executing the corresponding path in such graph. Our validation of the XACMET prototype implementation confirms the effectiveness of the proposed approach.
{"title":"An Automated Model-Based Test Oracle for Access Control Systems","authors":"A. Bertolino, Said Daoudagh, F. Lonetti, E. Marchetti","doi":"10.1145/3194733.3194743","DOIUrl":"https://doi.org/10.1145/3194733.3194743","url":null,"abstract":"In the context of XACML-based access control systems, an intensive testing activity is among the most adopted means to assure that sensible information or resources are correctly accessed. Unfortunately, it requires a huge effort for manual inspection of results: thus automated verdict derivation is a key aspect for improving the cost-effectiveness of testing. To this purpose, we introduce XACMET, a novel approach for automated model-based oracle definition. XACMET defines a typed graph, called the XAC-Graph, that models the XACML policy evaluation. The expected verdict of a specific request execution can thus be automatically derived by executing the corresponding path in such graph. Our validation of the XACMET prototype implementation confirms the effectiveness of the proposed approach.","PeriodicalId":423703,"journal":{"name":"2018 IEEE/ACM 13th International Workshop on Automation of Software Test (AST)","volume":"30 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2018-05-28","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"129787796","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Christof J. Budnik, M. Gario, Georgi A. Markov, Zhu Wang
Black-box software testing is a crucial part of quality assurance for industrial products. To verify the reliable behavior of software intensive systems, testing needs to ensure that the system produces the correct outputs from a variety of inputs. Even more critical, it needs to ensure that unexpected corner cases are tested. Existing approaches attempt to address this problem by the generation of input data to known outputs based on the domain knowledge of an expert. Such input space exploration, however, does not guarantee an adequate coverage of the output space as the test input data generation is done independently of the system output. The paper discusses a novel test case generation approach enabled by neural networks which promises higher probability of exposing system faults by systematically exploring the output space of the system under test. As such, the approach potentially improves the defect detection capability by identifying gaps in the test suite of uncovered system outputs. These gaps are closed by automatically determining inputs that lead to specic outputs by performing backward reasoning on an artificial neural network. The approach is demonstrated on an industrial train control system.
{"title":"Guided Test Case Generation through AI Enabled Output Space Exploration","authors":"Christof J. Budnik, M. Gario, Georgi A. Markov, Zhu Wang","doi":"10.1145/3194733.3194740","DOIUrl":"https://doi.org/10.1145/3194733.3194740","url":null,"abstract":"Black-box software testing is a crucial part of quality assurance for industrial products. To verify the reliable behavior of software intensive systems, testing needs to ensure that the system produces the correct outputs from a variety of inputs. Even more critical, it needs to ensure that unexpected corner cases are tested. Existing approaches attempt to address this problem by the generation of input data to known outputs based on the domain knowledge of an expert. Such input space exploration, however, does not guarantee an adequate coverage of the output space as the test input data generation is done independently of the system output. The paper discusses a novel test case generation approach enabled by neural networks which promises higher probability of exposing system faults by systematically exploring the output space of the system under test. As such, the approach potentially improves the defect detection capability by identifying gaps in the test suite of uncovered system outputs. These gaps are closed by automatically determining inputs that lead to specic outputs by performing backward reasoning on an artificial neural network. The approach is demonstrated on an industrial train control system.","PeriodicalId":423703,"journal":{"name":"2018 IEEE/ACM 13th International Workshop on Automation of Software Test (AST)","volume":"55 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2018-05-28","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"114905869","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Traditionally, people abstract away the infrastructure operation, such as power management, network traffic engineering and even the "cloud computing"? layers from software developers. This abstraction brings easier application development and maintenance but leads to complexities and inefficiencies for infrastructure operation. In this talk, we present our recent research into this area. Using data center power management, networking as well as cloud software management as examples.
{"title":"Towards Software-Defined and Self-Driving Cloud Infrastructure","authors":"W. Xu","doi":"10.1145/3194733.3194746","DOIUrl":"https://doi.org/10.1145/3194733.3194746","url":null,"abstract":"Traditionally, people abstract away the infrastructure operation, such as power management, network traffic engineering and even the \"cloud computing\"? layers from software developers. This abstraction brings easier application development and maintenance but leads to complexities and inefficiencies for infrastructure operation. In this talk, we present our recent research into this area. Using data center power management, networking as well as cloud software management as examples.","PeriodicalId":423703,"journal":{"name":"2018 IEEE/ACM 13th International Workshop on Automation of Software Test (AST)","volume":"80 3 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2018-05-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"127421898","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
京公网安备 11010802042870号
京ICP备2023020795号-1
扫码关注我们
求助内容：
应助结果提醒方式：