首页 > 最新文献

2018 IEEE/ACM 13th International Workshop on Automation of Software Test (AST)最新文献

英文 中文
Test Suite Reduction for Self-Organizing Systems: A Mutation-Based Approach 自组织系统的测试套件缩减:基于突变的方法
Pub Date : 2018-05-28 DOI: 10.1145/3194733.3194739
André Reichstaller, Benedikt Eberhardinger, Hella Ponsar, Alexander Knapp, W. Reif
We study regression testing and test suite reduction for self-organizing (SO) systems. The complex environments of SO systems typically require large test suites. The physical distribution of their components and their history-dependent behavior, however, make test execution very expensive. Consequently, an efficient test suite reduction mechanism is needed. The fundamental characteristic of SO systems is their ability to reconfigure themselves. We thus in- vestigate a mutation-based approach concentrating on reconfigura- tions, more specifically the communication between the distributed components in reconfigurations. Due to distribution, we argue for an explicit consideration of higher-order mutants and find a short- cut that makes the number of test cases to execute before reduction feasible. For the reduction task, we evaluate the applicability of two existing clustering techniques, Affinity Propagation and Dissimilar- ity-based Sparse Subset Selection. It turns out that these techniques are able to drastically reduce the original test suite while retaining a good mutation score. We discuss the approach by means of a test suite for a self-organizing production cell as a running example.
研究了自组织(SO)系统的回归测试和测试集缩减。SO系统的复杂环境通常需要大型测试套件。然而,它们组件的物理分布和依赖于历史的行为使得测试执行非常昂贵。因此,需要一个有效的测试套件缩减机制。SO系统的基本特征是它们能够重新配置自己。因此,我们研究了一种基于突变的方法,专注于重新配置,更具体地说,是重新配置中分布式组件之间的通信。由于分布,我们主张明确考虑高阶突变,并找到一条捷径,使得在减少之前执行的测试用例的数量可行。对于约简任务,我们评估了两种现有聚类技术的适用性,亲和传播和基于差异性的稀疏子集选择。事实证明,这些技术能够大幅减少原始测试套件,同时保持良好的突变得分。我们通过自组织生产单元的测试套件作为运行示例来讨论该方法。
{"title":"Test Suite Reduction for Self-Organizing Systems: A Mutation-Based Approach","authors":"André Reichstaller, Benedikt Eberhardinger, Hella Ponsar, Alexander Knapp, W. Reif","doi":"10.1145/3194733.3194739","DOIUrl":"https://doi.org/10.1145/3194733.3194739","url":null,"abstract":"We study regression testing and test suite reduction for self-organizing (SO) systems. The complex environments of SO systems typically require large test suites. The physical distribution of their components and their history-dependent behavior, however, make test execution very expensive. Consequently, an efficient test suite reduction mechanism is needed. The fundamental characteristic of SO systems is their ability to reconfigure themselves. We thus in- vestigate a mutation-based approach concentrating on reconfigura- tions, more specifically the communication between the distributed components in reconfigurations. Due to distribution, we argue for an explicit consideration of higher-order mutants and find a short- cut that makes the number of test cases to execute before reduction feasible. For the reduction task, we evaluate the applicability of two existing clustering techniques, Affinity Propagation and Dissimilar- ity-based Sparse Subset Selection. It turns out that these techniques are able to drastically reduce the original test suite while retaining a good mutation score. We discuss the approach by means of a test suite for a self-organizing production cell as a running example.","PeriodicalId":423703,"journal":{"name":"2018 IEEE/ACM 13th International Workshop on Automation of Software Test (AST)","volume":"78 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2018-05-28","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"125493484","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 7
Using Controlled Numbers of Real Faults and Mutants to Empirically Evaluate Coverage-Based Test Case Prioritization 使用控制数量的真实故障和突变来经验性地评估基于覆盖率的测试用例优先级
Pub Date : 2018-05-28 DOI: 10.1145/3194733.3194735
David Paterson, G. M. Kapfhammer, G. Fraser, Phil McMinn
Used to establish confidence in the correctness of evolving software, regression testing is an important, yet costly, task. Test case prioritization enables the rapid detection of faults during regression testing by reordering the test suite so that effective tests are run as early as is possible. However, a distinct lack of information about the regression faults found in complex real-world software forced prior experimental studies of these methods to use artificial faults called mutants. Using the Defects4J database of real faults, this paper presents the results of experiments evaluating the effectiveness of four representative test prioritization techniques. Since this paper’s results show that prioritization is susceptible to high amounts of variance when only one fault is present, our experiments also control the number of real faults and mutants in the program subject to regression testing. Our overall findings are that, in comparison to mutants, real faults are harder for reordered test suites to quickly detect, suggesting that mutants are not a surrogate for real faults.
回归测试用于建立对不断发展的软件正确性的信心,是一项重要但代价高昂的任务。通过重新排序测试套件,测试用例优先级可以在回归测试期间快速检测故障,以便尽可能早地运行有效的测试。然而,在复杂的现实世界软件中发现的回归错误的信息明显缺乏,迫使这些方法的先前实验研究使用称为突变的人工错误。利用缺陷4j真实故障数据库,给出了四种具有代表性的测试优先化技术的有效性评价实验结果。由于本文的结果表明,当只有一个故障存在时,优先级易受大量方差的影响,因此我们的实验还控制了回归测试中程序中实际故障和突变的数量。我们的总体发现是,与突变相比,重新排序的测试套件很难快速检测到真正的错误,这表明突变不是真正错误的替代品。
{"title":"Using Controlled Numbers of Real Faults and Mutants to Empirically Evaluate Coverage-Based Test Case Prioritization","authors":"David Paterson, G. M. Kapfhammer, G. Fraser, Phil McMinn","doi":"10.1145/3194733.3194735","DOIUrl":"https://doi.org/10.1145/3194733.3194735","url":null,"abstract":"Used to establish confidence in the correctness of evolving software, regression testing is an important, yet costly, task. Test case prioritization enables the rapid detection of faults during regression testing by reordering the test suite so that effective tests are run as early as is possible. However, a distinct lack of information about the regression faults found in complex real-world software forced prior experimental studies of these methods to use artificial faults called mutants. Using the Defects4J database of real faults, this paper presents the results of experiments evaluating the effectiveness of four representative test prioritization techniques. Since this paper’s results show that prioritization is susceptible to high amounts of variance when only one fault is present, our experiments also control the number of real faults and mutants in the program subject to regression testing. Our overall findings are that, in comparison to mutants, real faults are harder for reordered test suites to quickly detect, suggesting that mutants are not a surrogate for real faults.","PeriodicalId":423703,"journal":{"name":"2018 IEEE/ACM 13th International Workshop on Automation of Software Test (AST)","volume":"135 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2018-05-28","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"116544400","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 10
Software Testing as a Problem of Machine Learning: Towards a Foundation on Computational Learning Theory 软件测试作为机器学习的一个问题:迈向计算学习理论的基础
Pub Date : 2018-05-28 DOI: 10.1145/3194733.3194745
Hong Zhu
In recent years, the application of machine learning techniques to software testing has been an active research area. Among the most notable work reported in the literature are those experiments on the uses of supervised and semi-supervised learning techniques to develop test oracles so that the correctness of software outputs and behaviours on new test cases can be predicated. Experiment data show that it seems a promising approach to the test oracle automation problem. In general, software testing is an inductive inference in the course of which the tester attempts to deduce general properties of a software system by observing the behaviours of the system on a finite number of test cases. This talk discusses the theoretical foundation of software testing from the perspective of computational machine learning theories.
近年来,机器学习技术在软件测试中的应用一直是一个活跃的研究领域。在文献中报道的最值得注意的工作是那些使用监督和半监督学习技术来开发测试预言器的实验,以便可以预测新测试用例上软件输出和行为的正确性。实验数据表明,该方法是解决测试oracle自动化问题的有效方法。一般来说,软件测试是一个归纳推理的过程,在这个过程中,测试人员试图通过在有限数量的测试用例上观察系统的行为来推断软件系统的一般属性。本讲座从计算机器学习理论的角度探讨软件测试的理论基础。
{"title":"Software Testing as a Problem of Machine Learning: Towards a Foundation on Computational Learning Theory","authors":"Hong Zhu","doi":"10.1145/3194733.3194745","DOIUrl":"https://doi.org/10.1145/3194733.3194745","url":null,"abstract":"In recent years, the application of machine learning techniques to software testing has been an active research area. Among the most notable work reported in the literature are those experiments on the uses of supervised and semi-supervised learning techniques to develop test oracles so that the correctness of software outputs and behaviours on new test cases can be predicated. Experiment data show that it seems a promising approach to the test oracle automation problem. In general, software testing is an inductive inference in the course of which the tester attempts to deduce general properties of a software system by observing the behaviours of the system on a finite number of test cases. This talk discusses the theoretical foundation of software testing from the perspective of computational machine learning theories.","PeriodicalId":423703,"journal":{"name":"2018 IEEE/ACM 13th International Workshop on Automation of Software Test (AST)","volume":"53 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2018-05-28","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"133737383","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 3
Revisiting AI and Testing Methods to Infer FSM Models of Black-Box Systems 黑箱系统FSM模型的人工智能与测试方法研究
Pub Date : 2018-05-28 DOI: 10.1145/3194733.3194736
Roland Groz, A. Simão, N. Brémond, Catherine Oriat
Machine learning in the form of inference of state machine models has gained popularity in model-based testing as a means of retrieving models from software systems. By combining an old idea from machine inference with methods from automata testing in a heuristic approach, we propose a new promising direction for inferring black box systems that cannot be reset. Preliminary experiments show that this heuristic approach scales up well and outperforms more systematic approaches.
状态机模型推理形式的机器学习作为一种从软件系统中检索模型的手段,在基于模型的测试中得到了广泛的应用。通过将机器推理的旧思想与启发式方法中的自动机测试方法相结合,我们提出了一个新的有前途的方向来推断无法重置的黑匣子系统。初步实验表明,这种启发式方法可以很好地扩展,并且优于更系统的方法。
{"title":"Revisiting AI and Testing Methods to Infer FSM Models of Black-Box Systems","authors":"Roland Groz, A. Simão, N. Brémond, Catherine Oriat","doi":"10.1145/3194733.3194736","DOIUrl":"https://doi.org/10.1145/3194733.3194736","url":null,"abstract":"Machine learning in the form of inference of state machine models has gained popularity in model-based testing as a means of retrieving models from software systems. By combining an old idea from machine inference with methods from automata testing in a heuristic approach, we propose a new promising direction for inferring black box systems that cannot be reset. Preliminary experiments show that this heuristic approach scales up well and outperforms more systematic approaches.","PeriodicalId":423703,"journal":{"name":"2018 IEEE/ACM 13th International Workshop on Automation of Software Test (AST)","volume":"30 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2018-05-28","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"132415815","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 11
Planning-Based Security Testing of Web Applications 基于计划的Web应用程序安全测试
Pub Date : 2018-05-28 DOI: 10.1145/3194733.3194738
Josip Bozic, F. Wotawa
Web applications are deployed on machines around the globe and offer almost universal accessibility. The systems ensure functional interconnectivity between different components on a 24/7 basis. One of the most important requirements represents data confidentiality and secure authentication. However, implementation flaws and unfulfilled requirements can result in security leaks that can be eventually exploited by a malicious user. Here different testing methods are applied in order to detect software defects and prevent unauthorized access in advance. Automated planning and scheduling provides the possibility to specify a specific problem and to generate plans, which in turn guide the execution of a program. In this paper, a planning-based approach is introduced for modeling and testing of web applications. The specification offers a high degree of extendibility and configurability but overcomes the limits of traditional graphical representations as well. In this way, new testing possibilities emerge that eventually lead to better vulnerability detection, thereby ensuring more secure services.
Web应用程序部署在全球各地的机器上,并提供几乎普遍的可访问性。该系统可确保不同组件之间全天候的功能互联。最重要的需求之一是数据机密性和安全身份验证。然而,实现缺陷和未满足的需求可能导致安全漏洞,最终可能被恶意用户利用。这里采用了不同的测试方法,以便提前检测软件缺陷并防止未经授权的访问。自动计划和调度提供了指定特定问题和生成计划的可能性,这些计划反过来指导程序的执行。本文介绍了一种基于规划的web应用程序建模和测试方法。该规范提供了高度的可扩展性和可配置性,但也克服了传统图形表示的限制。通过这种方式,出现了新的测试可能性,最终导致更好的漏洞检测,从而确保更安全的服务。
{"title":"Planning-Based Security Testing of Web Applications","authors":"Josip Bozic, F. Wotawa","doi":"10.1145/3194733.3194738","DOIUrl":"https://doi.org/10.1145/3194733.3194738","url":null,"abstract":"Web applications are deployed on machines around the globe and offer almost universal accessibility. The systems ensure functional interconnectivity between different components on a 24/7 basis. One of the most important requirements represents data confidentiality and secure authentication. However, implementation flaws and unfulfilled requirements can result in security leaks that can be eventually exploited by a malicious user. Here different testing methods are applied in order to detect software defects and prevent unauthorized access in advance. Automated planning and scheduling provides the possibility to specify a specific problem and to generate plans, which in turn guide the execution of a program. In this paper, a planning-based approach is introduced for modeling and testing of web applications. The specification offers a high degree of extendibility and configurability but overcomes the limits of traditional graphical representations as well. In this way, new testing possibilities emerge that eventually lead to better vulnerability detection, thereby ensuring more secure services.","PeriodicalId":423703,"journal":{"name":"2018 IEEE/ACM 13th International Workshop on Automation of Software Test (AST)","volume":"3 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2018-05-28","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"123909410","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 8
Improving Continuous Integration with Similarity-Based Test Case Selection 使用基于相似度的测试用例选择改进持续集成
Pub Date : 2018-05-28 DOI: 10.1145/3194733.3194744
F. D. O. Neto, Azeem Ahmad, O. Leifler, K. Sandahl, Eduard Paul Enoiu
Automated testing is an essential component of Continuous Integration (CI) and Delivery (CD), such as scheduling automated test sessions on overnight builds. That allows stakeholders to execute entire test suites and achieve exhaustive test coverage, since running all tests is often infeasible during work hours, i.e., in parallel to development activities. On the other hand, developers also need test feedback from CI servers when pushing changes, even if not all test cases are executed. In this paper we evaluate similarity-based test case selection (SBTCS) on integration-level tests executed on continuous integration pipelines of two companies. We select test cases that maximise diversity of test coverage and reduce feedback time to developers. Our results confirm existing evidence that SBTCS is a strong candidate for test optimisation, by reducing feedback time (up to 92% faster in our case studies) while achieving full test coverage using only information from test artefacts themselves.
自动化测试是持续集成(CI)和交付(CD)的重要组成部分,例如在夜间构建中安排自动化测试会话。这允许涉众执行整个测试套件并实现详尽的测试覆盖,因为在工作时间内运行所有测试通常是不可行的,也就是说,与开发活动并行。另一方面,开发人员在推动更改时也需要来自CI服务器的测试反馈,即使不是所有的测试用例都被执行。在本文中,我们评估了基于相似性的测试用例选择(SBTCS)在两家公司的持续集成管道上执行的集成级测试。我们选择测试用例,使测试覆盖的多样性最大化,并减少对开发人员的反馈时间。我们的结果证实了现有的证据,即SBTCS是测试优化的一个强有力的候选者,通过减少反馈时间(在我们的案例研究中快了92%),同时仅使用来自测试工件本身的信息实现完整的测试覆盖。
{"title":"Improving Continuous Integration with Similarity-Based Test Case Selection","authors":"F. D. O. Neto, Azeem Ahmad, O. Leifler, K. Sandahl, Eduard Paul Enoiu","doi":"10.1145/3194733.3194744","DOIUrl":"https://doi.org/10.1145/3194733.3194744","url":null,"abstract":"Automated testing is an essential component of Continuous Integration (CI) and Delivery (CD), such as scheduling automated test sessions on overnight builds. That allows stakeholders to execute entire test suites and achieve exhaustive test coverage, since running all tests is often infeasible during work hours, i.e., in parallel to development activities. On the other hand, developers also need test feedback from CI servers when pushing changes, even if not all test cases are executed. In this paper we evaluate similarity-based test case selection (SBTCS) on integration-level tests executed on continuous integration pipelines of two companies. We select test cases that maximise diversity of test coverage and reduce feedback time to developers. Our results confirm existing evidence that SBTCS is a strong candidate for test optimisation, by reducing feedback time (up to 92% faster in our case studies) while achieving full test coverage using only information from test artefacts themselves.","PeriodicalId":423703,"journal":{"name":"2018 IEEE/ACM 13th International Workshop on Automation of Software Test (AST)","volume":"44 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2018-05-28","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"129507529","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 13
Memory Corruption Detecting Method Using Static Variables and Dynamic Memory Usage 基于静态变量和动态内存使用的内存损坏检测方法
Pub Date : 2018-05-28 DOI: 10.1145/3194733.3194741
Jihyun Park, Chang-Seo Park, Byoungju Choi, Gihun Chang
Memory fault detection has been continuously studied and various detection methods exist. However, there are still remains many memory defects that are difficult to debug. Memory corruption is one of those defects that often cause a system crash. However, there are many cases where the location of the crash is different from the actual location causing the actual memory corruption. These defects are difficult to solve by existing methods. In this paper, we propose a method to detect real time memory defects by using static global variables derived from execution binary file and dynamic memory usage obtained by tracing memory related functions. We implemented the proposed method as a tool and applied it to the application running on the IoTivity platform. Our tool detects defects very accurately with low overhead even for those whose detected location and the location of its cause are different.
内存故障检测一直是人们研究的热点,目前存在着各种检测方法。然而,仍然存在许多难以调试的内存缺陷。内存损坏是经常导致系统崩溃的缺陷之一。但是,在许多情况下,崩溃的位置与导致实际内存损坏的实际位置不同。这些缺陷是现有方法难以解决的。本文提出了一种利用从执行二进制文件中获得的静态全局变量和通过跟踪内存相关函数获得的动态内存使用情况来检测实时内存缺陷的方法。我们将提出的方法作为工具实现,并将其应用于运行在IoTivity平台上的应用程序。我们的工具以低开销非常准确地检测缺陷,即使对于那些检测位置和其原因的位置不同的缺陷也是如此。
{"title":"Memory Corruption Detecting Method Using Static Variables and Dynamic Memory Usage","authors":"Jihyun Park, Chang-Seo Park, Byoungju Choi, Gihun Chang","doi":"10.1145/3194733.3194741","DOIUrl":"https://doi.org/10.1145/3194733.3194741","url":null,"abstract":"Memory fault detection has been continuously studied and various detection methods exist. However, there are still remains many memory defects that are difficult to debug. Memory corruption is one of those defects that often cause a system crash. However, there are many cases where the location of the crash is different from the actual location causing the actual memory corruption. These defects are difficult to solve by existing methods. In this paper, we propose a method to detect real time memory defects by using static global variables derived from execution binary file and dynamic memory usage obtained by tracing memory related functions. We implemented the proposed method as a tool and applied it to the application running on the IoTivity platform. Our tool detects defects very accurately with low overhead even for those whose detected location and the location of its cause are different.","PeriodicalId":423703,"journal":{"name":"2018 IEEE/ACM 13th International Workshop on Automation of Software Test (AST)","volume":"2 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2018-05-28","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"131816940","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 1
On the Effectiveness of Random Testing for Android: Or How I Learned to Stop Worrying and Love the Monkey 关于Android随机测试的有效性:或者我如何学会停止担忧并爱上猴子
Pub Date : 2018-05-28 DOI: 10.1145/3194733.3194742
Priyam Patel, Gokul Srinivasan, Sydur Rahaman, Iulian Neamtiu
Random testing of Android apps is attractive due to ease-of-use and scalability, but its effectiveness could be questioned. Prior studies have shown that Monkey – a simple approach and tool for random testing of Android apps – is surprisingly effective, "beating" much more sophisticated tools by achieving higher coverage. We study how Monkey's parameters affect code coverage (at class, method, block, and line levels) and set out to answer several research questions centered around improving the effectiveness of Monkey-based random testing in Android, and how it compares with manual exploration. First, we show that random stress testing via Monkey is extremely efficient (85 seconds on average) and effective at crashing apps, including 15 widely-used apps that have millions (or even billions) of installs. Second, we vary Monkey's event distribution to change app behavior and measured the resulting coverage. We found that, except for isolated cases, altering Monkey's default event distribution is unlikely to lead to higher coverage. Third, we manually explore 62 apps and compare the resulting coverages; we found that coverage achieved via manual exploration is just 2-3% higher than that achieved via Monkey exploration. Finally, our analysis shows that coarse-grained coverage is highly indicative of fine-grained coverage, hence coarse-grained coverage (which imposes low collection overhead) hits a performance vs accuracy sweet spot.
Android应用的随机测试因其易用性和可扩展性而具有吸引力,但其有效性可能受到质疑。之前的研究表明,Monkey——一种用于随机测试Android应用程序的简单方法和工具——非常有效,通过获得更高的覆盖率,“击败”了更复杂的工具。我们研究了Monkey的参数如何影响代码覆盖率(在类,方法,块和行级别),并着手回答围绕提高Android中基于Monkey的随机测试的有效性的几个研究问题,以及它与手动探索的比较。首先,我们发现通过Monkey进行的随机压力测试非常有效(平均85秒)并且能够有效地让应用崩溃,包括15款拥有数百万(甚至数十亿)安装量的广泛使用应用。其次,我们改变Monkey的事件分布以改变应用行为,并测量结果覆盖率。我们发现,除了个别情况,改变Monkey的默认事件分布不太可能导致更高的覆盖率。第三,我们手动探索62个应用程序并比较结果覆盖率;我们发现,通过人工探索获得的覆盖率仅比通过Monkey探索获得的覆盖率高2-3%。最后,我们的分析表明,粗粒度覆盖高度指示细粒度覆盖,因此粗粒度覆盖(施加较低的收集开销)达到了性能与准确性的最佳点。
{"title":"On the Effectiveness of Random Testing for Android: Or How I Learned to Stop Worrying and Love the Monkey","authors":"Priyam Patel, Gokul Srinivasan, Sydur Rahaman, Iulian Neamtiu","doi":"10.1145/3194733.3194742","DOIUrl":"https://doi.org/10.1145/3194733.3194742","url":null,"abstract":"Random testing of Android apps is attractive due to ease-of-use and scalability, but its effectiveness could be questioned. Prior studies have shown that Monkey – a simple approach and tool for random testing of Android apps – is surprisingly effective, \"beating\" much more sophisticated tools by achieving higher coverage. We study how Monkey's parameters affect code coverage (at class, method, block, and line levels) and set out to answer several research questions centered around improving the effectiveness of Monkey-based random testing in Android, and how it compares with manual exploration. First, we show that random stress testing via Monkey is extremely efficient (85 seconds on average) and effective at crashing apps, including 15 widely-used apps that have millions (or even billions) of installs. Second, we vary Monkey's event distribution to change app behavior and measured the resulting coverage. We found that, except for isolated cases, altering Monkey's default event distribution is unlikely to lead to higher coverage. Third, we manually explore 62 apps and compare the resulting coverages; we found that coverage achieved via manual exploration is just 2-3% higher than that achieved via Monkey exploration. Finally, our analysis shows that coarse-grained coverage is highly indicative of fine-grained coverage, hence coarse-grained coverage (which imposes low collection overhead) hits a performance vs accuracy sweet spot.","PeriodicalId":423703,"journal":{"name":"2018 IEEE/ACM 13th International Workshop on Automation of Software Test (AST)","volume":"127 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2018-05-28","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"124222102","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 38
Testing Service Oriented Architectures Using Stateful Service Virtualization via Machine Learning 通过机器学习使用有状态服务虚拟化测试面向服务的架构
Pub Date : 2018-05-28 DOI: 10.1145/3194733.3194737
Hasan Ferit Eniser, A. Sen
Today's enterprise software systems are much complicated than the past. Increasing number of dependent applications, heterogeneous technologies and wide usage of Service Oriented Architectures (SOA), where numerous services communicate with each other, makes testing of such systems challenging. For testing these software systems, the concept of service virtualization is gaining popularity. Service virtualization is an automated technique to mimic the behavior of a given real service. Services can be classified as stateless or stateful services. Many services are stateful in nature. Although there are works in the literature for virtualization of stateless services, no such solution exists for stateful services. To the best of our knowledge, this is the first work for stateful service virtualization. We employ classification based and sequence-to-sequence based machine learning algorithms in developing our solutions. We demonstrate the validity of our approach on two data sets collected from real life services and obtain promising results.
今天的企业软件系统比过去复杂得多。越来越多的依赖应用程序、异构技术和面向服务的体系结构(SOA)的广泛使用(其中许多服务相互通信)使得此类系统的测试具有挑战性。为了测试这些软件系统,服务虚拟化的概念越来越流行。服务虚拟化是一种模拟给定真实服务行为的自动化技术。服务可以分为无状态服务和有状态服务。许多服务本质上是有状态的。尽管文献中有关于无状态服务虚拟化的工作,但没有针对有状态服务的解决方案。据我们所知,这是有状态服务虚拟化的第一个工作。我们采用基于分类和基于序列到序列的机器学习算法来开发我们的解决方案。我们在从现实生活服务中收集的两个数据集上证明了我们的方法的有效性,并获得了有希望的结果。
{"title":"Testing Service Oriented Architectures Using Stateful Service Virtualization via Machine Learning","authors":"Hasan Ferit Eniser, A. Sen","doi":"10.1145/3194733.3194737","DOIUrl":"https://doi.org/10.1145/3194733.3194737","url":null,"abstract":"Today's enterprise software systems are much complicated than the past. Increasing number of dependent applications, heterogeneous technologies and wide usage of Service Oriented Architectures (SOA), where numerous services communicate with each other, makes testing of such systems challenging. For testing these software systems, the concept of service virtualization is gaining popularity. Service virtualization is an automated technique to mimic the behavior of a given real service. Services can be classified as stateless or stateful services. Many services are stateful in nature. Although there are works in the literature for virtualization of stateless services, no such solution exists for stateful services. To the best of our knowledge, this is the first work for stateful service virtualization. We employ classification based and sequence-to-sequence based machine learning algorithms in developing our solutions. We demonstrate the validity of our approach on two data sets collected from real life services and obtain promising results.","PeriodicalId":423703,"journal":{"name":"2018 IEEE/ACM 13th International Workshop on Automation of Software Test (AST)","volume":"1 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2018-05-28","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"128566793","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 11
An Automated Model-Based Test Oracle for Access Control Systems 一种用于访问控制系统的基于模型的自动化测试Oracle
Pub Date : 2018-05-28 DOI: 10.1145/3194733.3194743
A. Bertolino, Said Daoudagh, F. Lonetti, E. Marchetti
In the context of XACML-based access control systems, an intensive testing activity is among the most adopted means to assure that sensible information or resources are correctly accessed. Unfortunately, it requires a huge effort for manual inspection of results: thus automated verdict derivation is a key aspect for improving the cost-effectiveness of testing. To this purpose, we introduce XACMET, a novel approach for automated model-based oracle definition. XACMET defines a typed graph, called the XAC-Graph, that models the XACML policy evaluation. The expected verdict of a specific request execution can thus be automatically derived by executing the corresponding path in such graph. Our validation of the XACMET prototype implementation confirms the effectiveness of the proposed approach.
在基于xacml的访问控制系统的上下文中,密集的测试活动是确保正确访问合理信息或资源的最常用方法之一。不幸的是,它需要大量的人工检查结果:因此,自动化的结论推导是提高测试成本效益的一个关键方面。为此,我们引入XACMET,这是一种基于模型的自动化oracle定义的新方法。XACMET定义了一个类型化图,称为XAC-Graph,它为XACML策略评估建模。通过执行图中相应的路径,可以自动得出特定请求执行的预期结果。我们对XACMET原型实现的验证证实了所提出方法的有效性。
{"title":"An Automated Model-Based Test Oracle for Access Control Systems","authors":"A. Bertolino, Said Daoudagh, F. Lonetti, E. Marchetti","doi":"10.1145/3194733.3194743","DOIUrl":"https://doi.org/10.1145/3194733.3194743","url":null,"abstract":"In the context of XACML-based access control systems, an intensive testing activity is among the most adopted means to assure that sensible information or resources are correctly accessed. Unfortunately, it requires a huge effort for manual inspection of results: thus automated verdict derivation is a key aspect for improving the cost-effectiveness of testing. To this purpose, we introduce XACMET, a novel approach for automated model-based oracle definition. XACMET defines a typed graph, called the XAC-Graph, that models the XACML policy evaluation. The expected verdict of a specific request execution can thus be automatically derived by executing the corresponding path in such graph. Our validation of the XACMET prototype implementation confirms the effectiveness of the proposed approach.","PeriodicalId":423703,"journal":{"name":"2018 IEEE/ACM 13th International Workshop on Automation of Software Test (AST)","volume":"30 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2018-05-28","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"129787796","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 10
期刊
2018 IEEE/ACM 13th International Workshop on Automation of Software Test (AST)
全部 Acc. Chem. Res. ACS Applied Bio Materials ACS Appl. Electron. Mater. ACS Appl. Energy Mater. ACS Appl. Mater. Interfaces ACS Appl. Nano Mater. ACS Appl. Polym. Mater. ACS BIOMATER-SCI ENG ACS Catal. ACS Cent. Sci. ACS Chem. Biol. ACS Chemical Health & Safety ACS Chem. Neurosci. ACS Comb. Sci. ACS Earth Space Chem. ACS Energy Lett. ACS Infect. Dis. ACS Macro Lett. ACS Mater. Lett. ACS Med. Chem. Lett. ACS Nano ACS Omega ACS Photonics ACS Sens. ACS Sustainable Chem. Eng. ACS Synth. Biol. Anal. Chem. BIOCHEMISTRY-US Bioconjugate Chem. BIOMACROMOLECULES Chem. Res. Toxicol. Chem. Rev. Chem. Mater. CRYST GROWTH DES ENERG FUEL Environ. Sci. Technol. Environ. Sci. Technol. Lett. Eur. J. Inorg. Chem. IND ENG CHEM RES Inorg. Chem. J. Agric. Food. Chem. J. Chem. Eng. Data J. Chem. Educ. J. Chem. Inf. Model. J. Chem. Theory Comput. J. Med. Chem. J. Nat. Prod. J PROTEOME RES J. Am. Chem. Soc. LANGMUIR MACROMOLECULES Mol. Pharmaceutics Nano Lett. Org. Lett. ORG PROCESS RES DEV ORGANOMETALLICS J. Org. Chem. J. Phys. Chem. J. Phys. Chem. A J. Phys. Chem. B J. Phys. Chem. C J. Phys. Chem. Lett. Analyst Anal. Methods Biomater. Sci. Catal. Sci. Technol. Chem. Commun. Chem. Soc. Rev. CHEM EDUC RES PRACT CRYSTENGCOMM Dalton Trans. Energy Environ. Sci. ENVIRON SCI-NANO ENVIRON SCI-PROC IMP ENVIRON SCI-WAT RES Faraday Discuss. Food Funct. Green Chem. Inorg. Chem. Front. Integr. Biol. J. Anal. At. Spectrom. J. Mater. Chem. A J. Mater. Chem. B J. Mater. Chem. C Lab Chip Mater. Chem. Front. Mater. Horiz. MEDCHEMCOMM Metallomics Mol. Biosyst. Mol. Syst. Des. Eng. Nanoscale Nanoscale Horiz. Nat. Prod. Rep. New J. Chem. Org. Biomol. Chem. Org. Chem. Front. PHOTOCH PHOTOBIO SCI PCCP Polym. Chem.
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
0
微信
客服QQ
Book学术公众号 扫码关注我们
反馈
×
意见反馈
请填写您的意见或建议
请填写您的手机或邮箱
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
现在去查看 取消
×
提示
确定
Book学术官方微信
Book学术文献互助
Book学术文献互助群
群 号:604180095
Book学术
文献互助 智能选刊 最新文献 互助须知 联系我们:info@booksci.cn
Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。
Copyright © 2023 Book学术 All rights reserved.
ghs 京公网安备 11010802042870号 京ICP备2023020795号-1