One of the grand challenges for adequately testing scientific software is due to the oracle problem. Metamorphic testing is the promise technique for addressing the problem through checking the satisfaction of a metamorphic relation that defines the correlation on the relation of paired test inputs and the one of their corresponding outputs. Therefore, checking the correctness of an individual test is replaced by checking the satisfaction of the metamorphic relation among a group of tests. However, the quality of the testing is highly depended on the identification of metamorphic relations. In this paper, we use an iterative approach for the development and refinement of metamorphic relations during testing process through testing an open source discrete dipole approximation program called ADDA. Through studying ADDA program, we discuss the experience and challenges of metamorphic testing for testing scientific software.
{"title":"An Application of Metamorphic Testing for Testing Scientific Software","authors":"Junhua Ding, Dongmei Zhang, Xin-Hua Hu","doi":"10.1145/2896971.2896981","DOIUrl":"https://doi.org/10.1145/2896971.2896981","url":null,"abstract":"One of the grand challenges for adequately testing scientific software is due to the oracle problem. Metamorphic testing is the promise technique for addressing the problem through checking the satisfaction of a metamorphic relation that defines the correlation on the relation of paired test inputs and the one of their corresponding outputs. Therefore, checking the correctness of an individual test is replaced by checking the satisfaction of the metamorphic relation among a group of tests. However, the quality of the testing is highly depended on the identification of metamorphic relations. In this paper, we use an iterative approach for the development and refinement of metamorphic relations during testing process through testing an open source discrete dipole approximation program called ADDA. Through studying ADDA program, we discuss the experience and challenges of metamorphic testing for testing scientific software.","PeriodicalId":221357,"journal":{"name":"2016 IEEE/ACM 1st International Workshop on Metamorphic Testing (MET)","volume":"123 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2016-05-14","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"122273823","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
We present strategies for metamorphic testing of compilers using opaque value injection, and experiences using the method to test compilers for the OpenGL shading language.
{"title":"Metamorphic Testing for (Graphics) Compilers","authors":"A. Donaldson, Andrei Lascu","doi":"10.1145/2896971.2896978","DOIUrl":"https://doi.org/10.1145/2896971.2896978","url":null,"abstract":"We present strategies for metamorphic testing of compilers using opaque value injection, and experiences using the method to test compilers for the OpenGL shading language.","PeriodicalId":221357,"journal":{"name":"2016 IEEE/ACM 1st International Workshop on Metamorphic Testing (MET)","volume":"27 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2016-05-14","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"126836206","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
When figuring out the expected output for each test case is difficult, metamorphic testing can be applied to alleviate such situations. An involved key challenge is to derive metamorphic relations for the program under test. This paper proposes a data-mutation directed metamorphic relation acquisition methodology called μMT. Experimental results on three case studies show that μMT is feasible in deriving metamorphic relations for numeric applications and the derived metamorphic relations show reasonable fault detection effectiveness.
{"title":"μMT: A Data Mutation Directed Metamorphic Relation Acquisition Methodology","authors":"Chang-ai Sun, Yiqiang Liu, Zuoyi Wang, W. Chan","doi":"10.1145/2896971.2896974","DOIUrl":"https://doi.org/10.1145/2896971.2896974","url":null,"abstract":"When figuring out the expected output for each test case is difficult, metamorphic testing can be applied to alleviate such situations. An involved key challenge is to derive metamorphic relations for the program under test. This paper proposes a data-mutation directed metamorphic relation acquisition methodology called μMT. Experimental results on three case studies show that μMT is feasible in deriving metamorphic relations for numeric applications and the derived metamorphic relations show reasonable fault detection effectiveness.","PeriodicalId":221357,"journal":{"name":"2016 IEEE/ACM 1st International Workshop on Metamorphic Testing (MET)","volume":"56 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2016-05-14","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"115273613","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Metamorphic testing uses domain-specific properties about a program’s intended behaviour to alleviate the oracle problem. From a given set of source test inputs, a set of follow- up test inputs are generated which have some relation to the source inputs, and their outputs are compared to outputs from the source tests, using metamorphic relations. We evaluate the use of an automated test input generation technique called dynamic symbolic execution (DSE) to generate the source test inputs for metamorphic testing. We investigate whether DSE increases source-code coverage and fault finding effectiveness of metamorphic testing compared to the use of random testing, and whether the use of metamorphic relations as a supportive technique improves the test inputs generated by DSE. Our results show that DSE improves the coverage and fault detection rate of metamorphic testing compared to random testing using significantly smaller test suites, and the use of metamorphic relations increases code coverage of both DSE and random tests considerably, but the improvement in the fault detection rate may be marginal and depends on the used metamorphic relations.
{"title":"Generating Source Inputs for Metamorphic Testing Using Dynamic Symbolic Execution","authors":"Eman Alatawi, Tim Miller, H. Søndergaard","doi":"10.1145/2896971.2896980","DOIUrl":"https://doi.org/10.1145/2896971.2896980","url":null,"abstract":"Metamorphic testing uses domain-specific properties about a program’s intended behaviour to alleviate the oracle problem. From a given set of source test inputs, a set of follow- up test inputs are generated which have some relation to the source inputs, and their outputs are compared to outputs from the source tests, using metamorphic relations. We evaluate the use of an automated test input generation technique called dynamic symbolic execution (DSE) to generate the source test inputs for metamorphic testing. We investigate whether DSE increases source-code coverage and fault finding effectiveness of metamorphic testing compared to the use of random testing, and whether the use of metamorphic relations as a supportive technique improves the test inputs generated by DSE. Our results show that DSE improves the coverage and fault detection rate of metamorphic testing compared to random testing using significantly smaller test suites, and the use of metamorphic relations increases code coverage of both DSE and random tests considerably, but the improvement in the fault detection rate may be marginal and depends on the used metamorphic relations.","PeriodicalId":221357,"journal":{"name":"2016 IEEE/ACM 1st International Workshop on Metamorphic Testing (MET)","volume":"1 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2016-05-14","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"115097312","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
A. Barus, T. Chen, Fei-Ching Kuo, Huai Liu, H. Schmidt
Metamorphic Testing (MT) aims to alleviate the oracle problem. In MT, testers define metamorphic relations (MRs) which are used to generate new test cases (referred to as follow-up test cases) from the available test cases (referred to as source test cases). Both source and follow-up test cases are executed and their outputs are verified against the relevant MRs, of which any violation implies that the software under test is faulty. So far, the research on the effectiveness of MT has been focused on the selection of better MRs (that is, MRs that are more likely to be violated). In addition to MR selection, the source and follow-up test cases may also affect the effectiveness of MT. Since follow-up test cases are defined by the source test cases and MRs, selection of source test cases will then affect the effectiveness of MT. However, in existing MT studies, random testing is commonly adopted as the test case selection strategy for source test cases. This study aims to investigate the impact of source test cases on the effectiveness of MT. Since Adaptive Random Testing (ART) has been developed as an enhancement to Random Testing (RT), this study will focus on comparing the performance of RT and ART as source test case selection strategies on the effectiveness of MT. Experiment results show that ART outperforms RT on enhancing the effectiveness of MT.
{"title":"The Impact of Source Test Case Selection on the Effectiveness of Metamorphic Testing","authors":"A. Barus, T. Chen, Fei-Ching Kuo, Huai Liu, H. Schmidt","doi":"10.1145/2896971.2896977","DOIUrl":"https://doi.org/10.1145/2896971.2896977","url":null,"abstract":"Metamorphic Testing (MT) aims to alleviate the oracle problem. In MT, testers define metamorphic relations (MRs) which are used to generate new test cases (referred to as follow-up test cases) from the available test cases (referred to as source test cases). Both source and follow-up test cases are executed and their outputs are verified against the relevant MRs, of which any violation implies that the software under test is faulty. So far, the research on the effectiveness of MT has been focused on the selection of better MRs (that is, MRs that are more likely to be violated). In addition to MR selection, the source and follow-up test cases may also affect the effectiveness of MT. Since follow-up test cases are defined by the source test cases and MRs, selection of source test cases will then affect the effectiveness of MT. However, in existing MT studies, random testing is commonly adopted as the test case selection strategy for source test cases. This study aims to investigate the impact of source test cases on the effectiveness of MT. Since Adaptive Random Testing (ART) has been developed as an enhancement to Random Testing (RT), this study will focus on comparing the performance of RT and ART as source test case selection strategies on the effectiveness of MT. Experiment results show that ART outperforms RT on enhancing the effectiveness of MT.","PeriodicalId":221357,"journal":{"name":"2016 IEEE/ACM 1st International Workshop on Metamorphic Testing (MET)","volume":"47 6 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2016-05-14","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"123268278","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Metamorphic Testing (MT) has been demonstrated to successfully alleviate oracle problems in many areas, including machine learning, compilers, bioinformatics, etc. However, given a new MT task, it is still very challenging to identify enough Metamorphic Relations (MRs) which are key components of MT. Aiming at this problem, we revisited previous MT applications and realized that they could form a very valuable database. Currently there is a lack of efficient link to get testers access these historical data. Therefore, in this paper, we propose to build METWiki, an MR repository, for collection and retrieval of these MRs. By providing exploration and search facilities, testers can find their desired MRs for reuse, reference, or inference. We also illustrate three sample usages of METWiki, which show important illuminations on how MRs can be obtained in practice.
{"title":"Looking for an MR? Try METWiki Today","authors":"Xiaoyuan Xie, Jiahao Li, Chen Wang, T. Chen","doi":"10.1145/2896971.2896976","DOIUrl":"https://doi.org/10.1145/2896971.2896976","url":null,"abstract":"Metamorphic Testing (MT) has been demonstrated to successfully alleviate oracle problems in many areas, including machine learning, compilers, bioinformatics, etc. However, given a new MT task, it is still very challenging to identify enough Metamorphic Relations (MRs) which are key components of MT. Aiming at this problem, we revisited previous MT applications and realized that they could form a very valuable database. Currently there is a lack of efficient link to get testers access these historical data. Therefore, in this paper, we propose to build METWiki, an MR repository, for collection and retrieval of these MRs. By providing exploration and search facilities, testers can find their desired MRs for reuse, reference, or inference. We also illustrate three sample usages of METWiki, which show important illuminations on how MRs can be obtained in practice.","PeriodicalId":221357,"journal":{"name":"2016 IEEE/ACM 1st International Workshop on Metamorphic Testing (MET)","volume":"4 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2016-05-14","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"121465000","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Michael Troup, A. Yang, A. Kamali, E. Giannoulatou, T. Chen, J. Ho
Testing of bioinformatics software often suffers from the oracle problem, especially when testing software that analyses human genome sequencing data. Metamorphic testing has been proposed to alleviate the oracle problem. Nonetheless, smaller research or clinical centres may be challenged by the complexity and resources required to implement a suitable metamorphic testing framework in practice. This paper presents a case study on how a cloud-based metamorphic testing framework can be applied to a widely used genomic sequencing pipeline, and discusses the future of implementing large-scale on-demand automated metamorphic testing using cloud-based resources.
{"title":"A Cloud-Based Framework for Applying Metamorphic Testing to a Bioinformatics Pipeline","authors":"Michael Troup, A. Yang, A. Kamali, E. Giannoulatou, T. Chen, J. Ho","doi":"10.1145/2896971.2896975","DOIUrl":"https://doi.org/10.1145/2896971.2896975","url":null,"abstract":"Testing of bioinformatics software often suffers from the oracle problem, especially when testing software that analyses human genome sequencing data. Metamorphic testing has been proposed to alleviate the oracle problem. Nonetheless, smaller research or clinical centres may be challenged by the complexity and resources required to implement a suitable metamorphic testing framework in practice. This paper presents a case study on how a cloud-based metamorphic testing framework can be applied to a widely used genomic sequencing pipeline, and discusses the future of implementing large-scale on-demand automated metamorphic testing using cloud-based resources.","PeriodicalId":221357,"journal":{"name":"2016 IEEE/ACM 1st International Workshop on Metamorphic Testing (MET)","volume":"9 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2016-05-14","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"128830498","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
M. Lindvall, D. Ganesan, Sigurthor Bjorgvinsson, K. Jónsson, Haukur Steinn Logason, Frederik Dietrich, Robert E. Wiegand
Testing becomes difficult when we cannot easily determine whether the system delivers the correct result or not. To address this issue, we have developed a framework for automated testing of NASA's DAT system using metamorphic testing principles combined with model based testing. Based on the results from using the framework to test DAT we have determined that this is a cost beneficial solution allowing for comprehensive testing that detects defects without having to develop complex testing infrastructure to determine the oracle. In our approach, DAT queries are automatically generated from a set of metamorphic testing models where each model encodes one or more of the identified equivalences. In addition, we use a scheme for generating time stamps that we use to automatically create time pairs that add another equivalence dimension.
{"title":"Agile Metamorphic Model-Based Testing","authors":"M. Lindvall, D. Ganesan, Sigurthor Bjorgvinsson, K. Jónsson, Haukur Steinn Logason, Frederik Dietrich, Robert E. Wiegand","doi":"10.1145/2896971.2896979","DOIUrl":"https://doi.org/10.1145/2896971.2896979","url":null,"abstract":"Testing becomes difficult when we cannot easily determine whether the system delivers the correct result or not. To address this issue, we have developed a framework for automated testing of NASA's DAT system using metamorphic testing principles combined with model based testing. Based on the results from using the framework to test DAT we have determined that this is a cost beneficial solution allowing for comprehensive testing that detects defects without having to develop complex testing infrastructure to determine the oracle. In our approach, DAT queries are automatically generated from a set of metamorphic testing models where each model encodes one or more of the identified equivalences. In addition, we use a scheme for generating time stamps that we use to automatically create time pairs that add another equivalence dimension.","PeriodicalId":221357,"journal":{"name":"2016 IEEE/ACM 1st International Workshop on Metamorphic Testing (MET)","volume":"1 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2016-05-14","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"131233539","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
京公网安备 11010802042870号
京ICP备2023020795号-1
扫码关注我们
求助内容：
应助结果提醒方式：