The results of empirical studies are limited to particular contexts, difficult to generalise and the studies themselves are expensive to perform. Despite these problems, empirical studies in software engineering can be made effective and they are important to both researchers and practitioners. The key to their effectiveness lies in the maximisation of the information that can be gained by examining existing studies, conducting power analyses for an accurate minimum sample size and benefiting from previous studies through replication. This approach was applied in a controlled experiment examining the combination of automated static analysis tools and code inspection in the context of verification and validation (V&V) of concurrent Java components. The combination of these V&V technologies was shown to be cost-effective despite the size of the study, which thus contributes to research in V&V technology evaluation.
{"title":"Maximising the information gained from an experimental analysis of code inspection and static analysis for concurrent java components","authors":"M. A. Wojcicki, P. Strooper","doi":"10.1145/1159733.1159761","DOIUrl":"https://doi.org/10.1145/1159733.1159761","url":null,"abstract":"The results of empirical studies are limited to particular contexts, difficult to generalise and the studies themselves are expensive to perform. Despite these problems, empirical studies in software engineering can be made effective and they are important to both researchers and practitioners. The key to their effectiveness lies in the maximisation of the information that can be gained by examining existing studies, conducting power analyses for an accurate minimum sample size and benefiting from previous studies through replication. This approach was applied in a controlled experiment examining the combination of automated static analysis tools and code inspection in the context of verification and validation (V&V) of concurrent Java components. The combination of these V&V technologies was shown to be cost-effective despite the size of the study, which thus contributes to research in V&V technology evaluation.","PeriodicalId":201305,"journal":{"name":"International Symposium on Empirical Software Engineering","volume":"8 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2006-09-21","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"116975464","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Five heuristics for attribute weighting in analogy-based effort estimation are evaluated in this paper. The baseline heuristic involves using all attributes with equal weights. We propose four additional heuristics that use rough set analysis for attribute weighting. These five heuristics are evaluated over five data sets related to software projects. Three of the data sets are publicly available, hence allowing comparison with other methods. The results indicate that three of the rough set analysis based heuristics perform better than the equal weights heuristic. This evaluation is based on an integrated measure of accuracy.
{"title":"A comparative study of attribute weighting heuristics for effort estimation by analogy","authors":"Jingzhou Li, G. Ruhe","doi":"10.1145/1159733.1159746","DOIUrl":"https://doi.org/10.1145/1159733.1159746","url":null,"abstract":"Five heuristics for attribute weighting in analogy-based effort estimation are evaluated in this paper. The baseline heuristic involves using all attributes with equal weights. We propose four additional heuristics that use rough set analysis for attribute weighting. These five heuristics are evaluated over five data sets related to software projects. Three of the data sets are publicly available, hence allowing comparison with other methods. The results indicate that three of the rough set analysis based heuristics perform better than the equal weights heuristic. This evaluation is based on an integrated measure of accuracy.","PeriodicalId":201305,"journal":{"name":"International Symposium on Empirical Software Engineering","volume":"15 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2006-09-21","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"124666490","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Testing with randomly generated test inputs, namely Random Testing, is a strategy that has been applied succefully in a lot of cases. Recently, some new adaptive approaches to the random generation of test cases have been proposed. Whereas there are many comparisons of Random Testing with Partition Testing, a systematic comparison of random testing techniques is still missing. This paper presents an empirical analysis and comparison of all random testing techniques from the field of Adaptive Random Testing (ART). The ART algorithms are compared for effectiveness using the mean F-measure, obtained through simulation and mutation analysis, and the P-measure. An interesting connection between the testing effectiveness measures F-measure and P-measure is described. The spatial distribution of test cases is determined to explain the behavior of the methods and identify possible shortcomings. Besides this, both the theoretical asymptotic runtime and the empirical runtime for each method are given.
{"title":"An empirical analysis and comparison of random testing techniques","authors":"Johannes Mayer, Christoph Schneckenburger","doi":"10.1145/1159733.1159751","DOIUrl":"https://doi.org/10.1145/1159733.1159751","url":null,"abstract":"Testing with randomly generated test inputs, namely Random Testing, is a strategy that has been applied succefully in a lot of cases. Recently, some new adaptive approaches to the random generation of test cases have been proposed. Whereas there are many comparisons of Random Testing with Partition Testing, a systematic comparison of random testing techniques is still missing. This paper presents an empirical analysis and comparison of all random testing techniques from the field of Adaptive Random Testing (ART). The ART algorithms are compared for effectiveness using the mean F-measure, obtained through simulation and mutation analysis, and the P-measure. An interesting connection between the testing effectiveness measures F-measure and P-measure is described. The spatial distribution of test cases is determined to explain the behavior of the methods and identify possible shortcomings. Besides this, both the theoretical asymptotic runtime and the empirical runtime for each method are given.","PeriodicalId":201305,"journal":{"name":"International Symposium on Empirical Software Engineering","volume":"22 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2006-09-21","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"127951601","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
This paper discusses software development using the Test Driven Development (TDD) methodology in two different environments (Windows and MSN divisions) at Microsoft. In both these case studies we measure the various context, product and outcome measures to compare and evaluate the efficacy of TDD. We observed a significant increase in quality of the code (greater than two times) for projects developed using TDD compared to similar projects developed in the same organization in a non-TDD fashion. The projects also took at least 15% extra upfront time for writing the tests. Additionally, the unit tests have served as auto documentation for the code when libraries/APIs had to be used as well as for code maintenance.
{"title":"Evaluating the efficacy of test-driven development: industrial case studies","authors":"Thirumalesh Bhat, Nachiappan Nagappan","doi":"10.1145/1159733.1159787","DOIUrl":"https://doi.org/10.1145/1159733.1159787","url":null,"abstract":"This paper discusses software development using the Test Driven Development (TDD) methodology in two different environments (Windows and MSN divisions) at Microsoft. In both these case studies we measure the various context, product and outcome measures to compare and evaluate the efficacy of TDD. We observed a significant increase in quality of the code (greater than two times) for projects developed using TDD compared to similar projects developed in the same organization in a non-TDD fashion. The projects also took at least 15% extra upfront time for writing the tests. Additionally, the unit tests have served as auto documentation for the code when libraries/APIs had to be used as well as for code maintenance.","PeriodicalId":201305,"journal":{"name":"International Symposium on Empirical Software Engineering","volume":"23 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2006-09-21","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"132226574","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Many software companies track and analyze project performance by measuring cost estimation accuracy. A high estimation error is frequently interpreted as poor estimation skills. This is not necessarily a correct interpretation. High estimation error can also be a result of other factors, such as high estimation complexity and insufficient cost control of the project. Through a real-life example we illustrate how the lack of proper estimation error analysis technique can bias analyses of cost estimation accuracy and lead to wrong conclusions. Further, we examine a selection of cost estimation studies, and show that they frequently do not take the necessary actions to ensure meaningful interpretations of estimation error data. Motivated by these results, we propose a general framework that, we believe, will improve analyses of software cost estimation error.
{"title":"A framework for the analysis of software cost estimation accuracy","authors":"Stein Grimstad, M. Jørgensen","doi":"10.1145/1159733.1159745","DOIUrl":"https://doi.org/10.1145/1159733.1159745","url":null,"abstract":"Many software companies track and analyze project performance by measuring cost estimation accuracy. A high estimation error is frequently interpreted as poor estimation skills. This is not necessarily a correct interpretation. High estimation error can also be a result of other factors, such as high estimation complexity and insufficient cost control of the project. Through a real-life example we illustrate how the lack of proper estimation error analysis technique can bias analyses of cost estimation accuracy and lead to wrong conclusions. Further, we examine a selection of cost estimation studies, and show that they frequently do not take the necessary actions to ensure meaningful interpretations of estimation error data. Motivated by these results, we propose a general framework that, we believe, will improve analyses of software cost estimation error.","PeriodicalId":201305,"journal":{"name":"International Symposium on Empirical Software Engineering","volume":"1 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2006-09-21","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"130961683","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Scenario-based methods for evaluating software architecture require a large number of stakeholders to be collocated for evaluation sessions. Collocating stakeholders is often an expensive exercise. To reduce expense, we have proposed a framework for supporting software architecture evaluation process using groupware systems. This paper presents a controlled experiment that we conducted to assess the effectiveness of scenario profile construction using distributed meetings. We used a cross-over experiment involving 32 teams of three 3rd and 4th year undergraduate students. We found that the quality of scenarios produced by distributed teams using a groupware tool were significantly better than the quality of scenarios produced by face-to-face teams (p<0.001). However, questionnaires indicated that most participants preferred the face-toface arrangement (82%) and 60% thought the distributed meetings were less efficient. We conclude that distributed meetings are extremely effective but that tool support must be of a high standard or participants will not find distributed meetings acceptable.
{"title":"Distributed versus face-to-face meetings for architecture evalution: a controlled experiment","authors":"M. Babar, B. Kitchenham, D. R. Jeffery","doi":"10.1145/1159733.1159771","DOIUrl":"https://doi.org/10.1145/1159733.1159771","url":null,"abstract":"Scenario-based methods for evaluating software architecture require a large number of stakeholders to be collocated for evaluation sessions. Collocating stakeholders is often an expensive exercise. To reduce expense, we have proposed a framework for supporting software architecture evaluation process using groupware systems. This paper presents a controlled experiment that we conducted to assess the effectiveness of scenario profile construction using distributed meetings. We used a cross-over experiment involving 32 teams of three 3rd and 4th year undergraduate students. We found that the quality of scenarios produced by distributed teams using a groupware tool were significantly better than the quality of scenarios produced by face-to-face teams (p<0.001). However, questionnaires indicated that most participants preferred the face-toface arrangement (82%) and 60% thought the distributed meetings were less efficient. We conclude that distributed meetings are extremely effective but that tool support must be of a high standard or participants will not find distributed meetings acceptable.","PeriodicalId":201305,"journal":{"name":"International Symposium on Empirical Software Engineering","volume":"40 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2006-09-21","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"127411941","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
S. Vegas, Natalia Juristo Juzgado, A. Moreno, Martín Solari, P. Letelier
The replication of experiments is a key undertaking in SE. Successful replications enable a discipline's body of knowledge to grow, as the results are added to those of earlier replications. However, replication is extremely difficult in SE, primarily because it is difficult to get a setting that is exactly the same as in the original experiment. Consequently, changes have to be made to the experiment to adapt it to the new site. To be able to replicate an experiment, information also has to be transmitted (usually orally and in writing) between the researchers who ran the experiment earlier and the ones who are going to replicate the experiment. This article examines the influence of the type of communication there is between experimenters on how successful a replication is. We have studied three replications of the same experiment in which different types of communication were used.
{"title":"Analysis of the influence of communication between researchers on experiment replication","authors":"S. Vegas, Natalia Juristo Juzgado, A. Moreno, Martín Solari, P. Letelier","doi":"10.1145/1159733.1159741","DOIUrl":"https://doi.org/10.1145/1159733.1159741","url":null,"abstract":"The replication of experiments is a key undertaking in SE. Successful replications enable a discipline's body of knowledge to grow, as the results are added to those of earlier replications. However, replication is extremely difficult in SE, primarily because it is difficult to get a setting that is exactly the same as in the original experiment. Consequently, changes have to be made to the experiment to adapt it to the new site. To be able to replicate an experiment, information also has to be transmitted (usually orally and in writing) between the researchers who ran the experiment earlier and the ones who are going to replicate the experiment. This article examines the influence of the type of communication there is between experimenters on how successful a replication is. We have studied three replications of the same experiment in which different types of communication were used.","PeriodicalId":201305,"journal":{"name":"International Symposium on Empirical Software Engineering","volume":"32 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2006-09-21","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"125867758","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
B. Kitchenham, H. Al-Kilidar, M. Babar, Michael Berry, Karl Cox, J. Keung, F. Kurniawati, M. Staples, He Zhang, Liming Zhu
Background. Several researchers have criticized the standards of performing and reporting empirical studies in software engineering. In order to address this problem, Andreas Jedlitschka and Dietmar Pfahl have produced reporting guidelines for controlled experiments in software engineering. They pointed out that their guidelines needed evaluation. We agree that guidelines need to be evaluated before they can be widely adopted. If guidelines are flawed, they will cause more problems that they solve.Aim. The aim of this paper is to present the method we used to evaluate the guidelines and report the results of our evaluation exercise. We suggest our evaluation process may be of more general use if reporting guidelines for other types of empirical study are developed.Method. We used perspective-based inspections to perform a theoretical evaluation of the guidelines. A separate inspection was performed for each perspective. The perspectives used were: Researcher, Practitioner/Consultant, Meta-analyst, Replicator, Reviewer and Author. Apart from the Author perspective, the inspections were based on a set of questions derived by brainstorming. The inspection using the Author perspective reviewed each section of the guidelines sequentially. Results. The question-based perspective inspections detected 42 issues where the guidelines would benefit from amendment or clarification and 8 defects.Conclusions. Reporting guidelines need to specify what information goes into what section and avoid excessive duplication. Software engineering researchers need to be cautious about adopting reporting guidelines that differ from those used by other disciplines. The current guidelines need to be revised and the revised guidelines need to be subjected to further theoretical and empirical validation. Perspective-based inspection is a useful validation method but the practitioner/consultant perspective presents difficulties.
{"title":"Evaluating guidelines for empirical software engineering studies","authors":"B. Kitchenham, H. Al-Kilidar, M. Babar, Michael Berry, Karl Cox, J. Keung, F. Kurniawati, M. Staples, He Zhang, Liming Zhu","doi":"10.1145/1159733.1159742","DOIUrl":"https://doi.org/10.1145/1159733.1159742","url":null,"abstract":"Background. Several researchers have criticized the standards of performing and reporting empirical studies in software engineering. In order to address this problem, Andreas Jedlitschka and Dietmar Pfahl have produced reporting guidelines for controlled experiments in software engineering. They pointed out that their guidelines needed evaluation. We agree that guidelines need to be evaluated before they can be widely adopted. If guidelines are flawed, they will cause more problems that they solve.Aim. The aim of this paper is to present the method we used to evaluate the guidelines and report the results of our evaluation exercise. We suggest our evaluation process may be of more general use if reporting guidelines for other types of empirical study are developed.Method. We used perspective-based inspections to perform a theoretical evaluation of the guidelines. A separate inspection was performed for each perspective. The perspectives used were: Researcher, Practitioner/Consultant, Meta-analyst, Replicator, Reviewer and Author. Apart from the Author perspective, the inspections were based on a set of questions derived by brainstorming. The inspection using the Author perspective reviewed each section of the guidelines sequentially. Results. The question-based perspective inspections detected 42 issues where the guidelines would benefit from amendment or clarification and 8 defects.Conclusions. Reporting guidelines need to specify what information goes into what section and avoid excessive duplication. Software engineering researchers need to be cautious about adopting reporting guidelines that differ from those used by other disciplines. The current guidelines need to be revised and the revised guidelines need to be subjected to further theoretical and empirical validation. Perspective-based inspection is a useful validation method but the practitioner/consultant perspective presents difficulties.","PeriodicalId":201305,"journal":{"name":"International Symposium on Empirical Software Engineering","volume":"110 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2006-09-21","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"115642781","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
The objective of this qualitative study was to understand the complex practice of software testing, and based on this knowledge, to develop process improvement propositions that could concurrently reduce development and testing costs and improve software quality. First, a survey of testing practices was onducted and 26 organizational units (OUs) were interviewed. From this sample, five OUs were further selected for an in-depth case study. The study used grounded theory as its research method and the data was collected from 41 theme-based interviews. The analysis yielded improvement propositions that included enhanced testability of software components, efficient communication and interaction between development and testing, early involvement of testing, and risk-based testing. The connective and central improvement proposition was that testing ought to adapt to the business orientation of the OU. Other propositions were integrated around this central proposition. The results of this study can be used in improving development and testing processes.
{"title":"Improving software testing by observing practice","authors":"Ossi Taipale, K. Smolander","doi":"10.1145/1159733.1159773","DOIUrl":"https://doi.org/10.1145/1159733.1159773","url":null,"abstract":"The objective of this qualitative study was to understand the complex practice of software testing, and based on this knowledge, to develop process improvement propositions that could concurrently reduce development and testing costs and improve software quality. First, a survey of testing practices was onducted and 26 organizational units (OUs) were interviewed. From this sample, five OUs were further selected for an in-depth case study. The study used grounded theory as its research method and the data was collected from 41 theme-based interviews. The analysis yielded improvement propositions that included enhanced testability of software components, efficient communication and interaction between development and testing, early involvement of testing, and risk-based testing. The connective and central improvement proposition was that testing ought to adapt to the business orientation of the OU. Other propositions were integrated around this central proposition. The results of this study can be used in improving development and testing processes.","PeriodicalId":201305,"journal":{"name":"International Symposium on Empirical Software Engineering","volume":"9 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2006-09-21","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"122448074","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Replication of experiments, or performing a series of related studies, aims at attaining a higher level of validity of results. This paper reports on a series of empirical studies devoted to comparing informal release planning with two variants of optimization-based release planning.Two research questions were studied: How does optimization-based release planning compare with informal planning in terms of (i) time to generate release plans, and the feasibility and quality of those plans, and (ii) understanding and confidence of generated solutions and trust in the release planning process.For the family of empirical studies, the paper presents two types of results related to (i) the two research questions to compare the release planning techniques, and (ii) the evolution and lessons learned while conducting the studies.
{"title":"A family of empirical studies to compare informal and optimization-based planning of software releases","authors":"Gengshen Du, J. McElroy, G. Ruhe","doi":"10.1145/1159733.1159766","DOIUrl":"https://doi.org/10.1145/1159733.1159766","url":null,"abstract":"Replication of experiments, or performing a series of related studies, aims at attaining a higher level of validity of results. This paper reports on a series of empirical studies devoted to comparing informal release planning with two variants of optimization-based release planning.Two research questions were studied: How does optimization-based release planning compare with informal planning in terms of (i) time to generate release plans, and the feasibility and quality of those plans, and (ii) understanding and confidence of generated solutions and trust in the release planning process.For the family of empirical studies, the paper presents two types of results related to (i) the two research questions to compare the release planning techniques, and (ii) the evolution and lessons learned while conducting the studies.","PeriodicalId":201305,"journal":{"name":"International Symposium on Empirical Software Engineering","volume":"348 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2006-09-21","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"124296553","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
京公网安备 11010802042870号
京ICP备2023020795号-1
扫码关注我们
求助内容：
应助结果提醒方式：