Context: The difficulty (not just effort) of obtaining access for software engineering empirical studies in industry varies greatly. Supposedly, some of this variance in difficulty is particular, stemming from properties of individual contexts (the industrial partners and their work), while the rest is repeatable, related to properties of the research question and research design. Question: What are these recurring difficulty factors that arise from research question and research design? What mechanisms produce their influence? Method: We use ideation and knowledge extraction from research experience to identify potential difficulty factors, use expert discussion to understand their mechanisms, and use concept analysis to arrange them into a taxonomy. We evaluate the result by comparatively applying it to two research efforts pursued by the same research group. Results: We find six scope factors, five problematic intervention effects factors, and seven helpful intervention (side-)effects factors. Conclusion: Considering these factors systematically during the formulation of a research question and the design of a research method can help with balancing data collection difficulty with results validity and relevance.
{"title":"Difficulty Factors of Obtaining Access for Empirical Studies in Industry","authors":"L. Prechelt, Franz Zieris, H. Schmeisky","doi":"10.1109/CESI.2015.11","DOIUrl":"https://doi.org/10.1109/CESI.2015.11","url":null,"abstract":"Context: The difficulty (not just effort) of obtaining access for software engineering empirical studies in industry varies greatly. Supposedly, some of this variance in difficulty is particular, stemming from properties of individual contexts (the industrial partners and their work), while the rest is repeatable, related to properties of the research question and research design. Question: What are these recurring difficulty factors that arise from research question and research design? What mechanisms produce their influence? Method: We use ideation and knowledge extraction from research experience to identify potential difficulty factors, use expert discussion to understand their mechanisms, and use concept analysis to arrange them into a taxonomy. We evaluate the result by comparatively applying it to two research efforts pursued by the same research group. Results: We find six scope factors, five problematic intervention effects factors, and seven helpful intervention (side-)effects factors. Conclusion: Considering these factors systematically during the formulation of a research question and the design of a research method can help with balancing data collection difficulty with results validity and relevance.","PeriodicalId":222668,"journal":{"name":"2015 IEEE/ACM 3rd International Workshop on Conducting Empirical Studies in Industry","volume":"11 6 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2015-05-16","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"114183271","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Conducting empirical studies and transferring their results into industry in a design discipline such as software engineering is ambitious. This is due to contextual restrictions, representativeness as well as problems in aggregating results from individual studies towards guidelines for practitioners. Nevertheless, they are necessary, as scientific contributions need to be challengeable. Significant progress in areas such as measurement, controlled experiments, industrial case studies, empirical based modeling, and packaging knowledge have been made over the past 30 to 40 years. External visibility has been increased significantly by means of books, conferences & journals! Future challenges include attracting more industrial contributions to the existing body of knowledge, using quantitative & qualitative studies to create more trustful evidences, and aggregation of empirical results. These challenges require community efforts.
{"title":"The Maturation of Empirical Studies","authors":"H. D. Rombach, Andreas Jedlitschka","doi":"10.1109/CESI.2015.7","DOIUrl":"https://doi.org/10.1109/CESI.2015.7","url":null,"abstract":"Conducting empirical studies and transferring their results into industry in a design discipline such as software engineering is ambitious. This is due to contextual restrictions, representativeness as well as problems in aggregating results from individual studies towards guidelines for practitioners. Nevertheless, they are necessary, as scientific contributions need to be challengeable. Significant progress in areas such as measurement, controlled experiments, industrial case studies, empirical based modeling, and packaging knowledge have been made over the past 30 to 40 years. External visibility has been increased significantly by means of books, conferences & journals! Future challenges include attracting more industrial contributions to the existing body of knowledge, using quantitative & qualitative studies to create more trustful evidences, and aggregation of empirical results. These challenges require community efforts.","PeriodicalId":222668,"journal":{"name":"2015 IEEE/ACM 3rd International Workshop on Conducting Empirical Studies in Industry","volume":"21 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2015-05-16","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"134113954","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Convincing industrial partners to support an exploratory study can be difficult, as benefits are often fuzzy at the beginning. The objective of this paper is to present recommendations for industrial exploratory studies based on our experience. The recommendations are based on ten months of observations during a non-participant, exploratory study with a single industrial partner. This study confirms a number of methodological challenges already identified in the software engineering literature. Based on recommendations from the literature and our own experience, we propose a process for future observational exploratory studies.
{"title":"Planning for the Unknown: Lessons Learned from Ten Months of Non-participant Exploratory Observations in the Industry","authors":"Mathieu Lavallée, P. Robillard","doi":"10.1109/CESI.2015.10","DOIUrl":"https://doi.org/10.1109/CESI.2015.10","url":null,"abstract":"Convincing industrial partners to support an exploratory study can be difficult, as benefits are often fuzzy at the beginning. The objective of this paper is to present recommendations for industrial exploratory studies based on our experience. The recommendations are based on ten months of observations during a non-participant, exploratory study with a single industrial partner. This study confirms a number of methodological challenges already identified in the software engineering literature. Based on recommendations from the literature and our own experience, we propose a process for future observational exploratory studies.","PeriodicalId":222668,"journal":{"name":"2015 IEEE/ACM 3rd International Workshop on Conducting Empirical Studies in Industry","volume":"77 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2015-05-16","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"125186382","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
What is an ideal collaboration from a research perspective? An archetypical researcher would answer:aligns perfectly with existing research directions brings in revenue from industry provides the opportunity to cite blue-chip company use of research methods and tools generates many conference papers. What is an ideal collaboration from an industry perspective? An archetypical practitioner would answer:aligns perfectly with real project needs provides a good return on investment enhances and protects company Intellectual Property increases the capability of the organisation. These objectives are at best in tension and at worst in direct conflict. So, what is the common ground? Is it possible to reach a win-win situation? This presentation will seek to answer these questions, to provide examples of "good" and "bad" collaborations and to suggest some lessons learned.
{"title":"I'll Tell You What I Want, What I Really, Really Want: An Industry Perspective on the Effective Application of Research in Projects","authors":"Alistair Mavin","doi":"10.1109/CESI.2015.16","DOIUrl":"https://doi.org/10.1109/CESI.2015.16","url":null,"abstract":"What is an ideal collaboration from a research perspective? An archetypical researcher would answer:aligns perfectly with existing research directions brings in revenue from industry provides the opportunity to cite blue-chip company use of research methods and tools generates many conference papers. What is an ideal collaboration from an industry perspective? An archetypical practitioner would answer:aligns perfectly with real project needs provides a good return on investment enhances and protects company Intellectual Property increases the capability of the organisation. These objectives are at best in tension and at worst in direct conflict. So, what is the common ground? Is it possible to reach a win-win situation? This presentation will seek to answer these questions, to provide examples of \"good\" and \"bad\" collaborations and to suggest some lessons learned.","PeriodicalId":222668,"journal":{"name":"2015 IEEE/ACM 3rd International Workshop on Conducting Empirical Studies in Industry","volume":"1 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2015-05-16","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"115923648","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Empirical research studies are the principal mechanism through which the software engineering research community studies and learns from software engineering practice. The focus on empirical studies has increased significantly in the past decade, more or less coinciding with the emergence of evidence-based software engineering, an idea that was proposed in 2004. As a consequence, the software engineering community is familiar with a range of empirical methods. However, while several overviews exist of popular empirical research methods, such as case studies and experiments, we lack a 'holistic' view of a more complete spectrum of research methods. Furthermore, while researchers will readily accept that all methods have inherent limitations, methods such as case study are still frequently critiqued for the lack of control that a researcher can exert in such a study, their use of qualitative data, and the limited generalizability that can be achieved. Controlled experiments are seen by many as yielding stronger evidence than case studies, but these can also be criticized due to the limited realism of the context in which they are conducted. We identify a holistic set of research methods and indicate their strengths and weaknesses in relation to various research elements.
{"title":"A Holistic Overview of Software Engineering Research Strategies","authors":"Klaas-Jan Stol, Brian Fitzgerald","doi":"10.1109/CESI.2015.15","DOIUrl":"https://doi.org/10.1109/CESI.2015.15","url":null,"abstract":"Empirical research studies are the principal mechanism through which the software engineering research community studies and learns from software engineering practice. The focus on empirical studies has increased significantly in the past decade, more or less coinciding with the emergence of evidence-based software engineering, an idea that was proposed in 2004. As a consequence, the software engineering community is familiar with a range of empirical methods. However, while several overviews exist of popular empirical research methods, such as case studies and experiments, we lack a 'holistic' view of a more complete spectrum of research methods. Furthermore, while researchers will readily accept that all methods have inherent limitations, methods such as case study are still frequently critiqued for the lack of control that a researcher can exert in such a study, their use of qualitative data, and the limited generalizability that can be achieved. Controlled experiments are seen by many as yielding stronger evidence than case studies, but these can also be criticized due to the limited realism of the context in which they are conducted. We identify a holistic set of research methods and indicate their strengths and weaknesses in relation to various research elements.","PeriodicalId":222668,"journal":{"name":"2015 IEEE/ACM 3rd International Workshop on Conducting Empirical Studies in Industry","volume":"1 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2015-05-16","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"125886156","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
CONTEXT. Alignment is a key factor for success in many software development projects. Aligned teams are capable of bringing collaboration and positive results to companies; whereas misalignment among developers can make a conflicted environment and even lead the project to failure. OBJECTIVE. To assist developers in an embedded software development company in their conceptual alignment regarding source code quality. METHOD. In the organizational context, plan and perform a series of studies such as surveys, systematic literature review (SLR), qualitative data analysis and focus group to support the identification of conceptual misalignments among developers and establish common terminology and guidance concerning source code quality. RESULTS. The results from a survey conducted in one company showed a conceptual misalignment among developers regarding the source code quality that was triggering continuous rework during software evolution activities. Through an SLR and a qualitative analysis of code snippets, a set of evidence-based coding guidelines for readability and understandability of source code were formulated. These guidelines were evaluated and used as an instrument for aligning source code perspectives during a focus group, showing their feasibility and adequacy to the company's context. CONCLUSIONS. The use of all contextual information observed - e.g. teams' locations, software development context, and time constraints - along with the information gathered during the industry-academia collaboration was particularly important to help us appropriately chose research methods to be used, and formulate evidence-based coding guidelines that matched the company's needs and expectations. Further evaluations have to be carried out to ensure the quality impact of some guidelines proposed before using them all over the company.
{"title":"On the Alignment of Source Code Quality Perspectives through Experimentation: An Industrial Case","authors":"Talita Vieira Ribeiro, G. Travassos","doi":"10.1109/CESI.2015.12","DOIUrl":"https://doi.org/10.1109/CESI.2015.12","url":null,"abstract":"CONTEXT. Alignment is a key factor for success in many software development projects. Aligned teams are capable of bringing collaboration and positive results to companies; whereas misalignment among developers can make a conflicted environment and even lead the project to failure. OBJECTIVE. To assist developers in an embedded software development company in their conceptual alignment regarding source code quality. METHOD. In the organizational context, plan and perform a series of studies such as surveys, systematic literature review (SLR), qualitative data analysis and focus group to support the identification of conceptual misalignments among developers and establish common terminology and guidance concerning source code quality. RESULTS. The results from a survey conducted in one company showed a conceptual misalignment among developers regarding the source code quality that was triggering continuous rework during software evolution activities. Through an SLR and a qualitative analysis of code snippets, a set of evidence-based coding guidelines for readability and understandability of source code were formulated. These guidelines were evaluated and used as an instrument for aligning source code perspectives during a focus group, showing their feasibility and adequacy to the company's context. CONCLUSIONS. The use of all contextual information observed - e.g. teams' locations, software development context, and time constraints - along with the information gathered during the industry-academia collaboration was particularly important to help us appropriately chose research methods to be used, and formulate evidence-based coding guidelines that matched the company's needs and expectations. Further evaluations have to be carried out to ensure the quality impact of some guidelines proposed before using them all over the company.","PeriodicalId":222668,"journal":{"name":"2015 IEEE/ACM 3rd International Workshop on Conducting Empirical Studies in Industry","volume":"15 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2015-05-16","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"133307845","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
S. Vegas, Óscar Dieste Tubío, Natalia Juristo Juzgado
Controlled experiments in laboratory settings are relatively commonplace in software engineering, but experiments in industry are thin on the ground. Of the few existing cases, most are 1-1 (running one experiment at one company), just a few are n-1 (running n experiments at one company) and still fewer are 1-n (running one and the same experiment at n companies). In this paper we report the difficulties that we experienced running the same experiment at several companies. We ran the same experiment in five different settings at three companies, and the results were transferred to each company so that they could exploit the resulting evidence in their decision-making process. We have found that: 1) it was relatively easy to get companies involved; 2) they did not cooperate as much as they had agreed to in the project proposal; 3) our industrial environments imposed many more constraints on the experimental design than laboratory environments; 4) professionals were less motivated than students; 5) the reliability of the results could be compromised by subject characteristics and behaviour; and 6) experiment findings could not be transferred using just the standard reporting guidelines that are used for scientific articles.
{"title":"Difficulties in Running Experiments in the Software Industry: Experiences from the Trenches","authors":"S. Vegas, Óscar Dieste Tubío, Natalia Juristo Juzgado","doi":"10.5555/2819303.2819307","DOIUrl":"https://doi.org/10.5555/2819303.2819307","url":null,"abstract":"Controlled experiments in laboratory settings are relatively commonplace in software engineering, but experiments in industry are thin on the ground. Of the few existing cases, most are 1-1 (running one experiment at one company), just a few are n-1 (running n experiments at one company) and still fewer are 1-n (running one and the same experiment at n companies). In this paper we report the difficulties that we experienced running the same experiment at several companies. We ran the same experiment in five different settings at three companies, and the results were transferred to each company so that they could exploit the resulting evidence in their decision-making process. We have found that: 1) it was relatively easy to get companies involved; 2) they did not cooperate as much as they had agreed to in the project proposal; 3) our industrial environments imposed many more constraints on the experimental design than laboratory environments; 4) professionals were less motivated than students; 5) the reliability of the results could be compromised by subject characteristics and behaviour; and 6) experiment findings could not be transferred using just the standard reporting guidelines that are used for scientific articles.","PeriodicalId":222668,"journal":{"name":"2015 IEEE/ACM 3rd International Workshop on Conducting Empirical Studies in Industry","volume":"27 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2015-05-16","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"121862832","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
The field of Empirical Software Engineering has undergone a much-needed expansion the last decade, and papers of all shapes and sizes are more or less mandated to have an "empirical" part to be published in premiere venues. The positive trend has researchers realizing the benefits, but also the investments needed, inherent to industry collaboration. That is, real practitioners, involved in the development of software intensive product, system, and service development. This paper shortly summarizes lessons learned from over ten years experience of industrial collaboration, and knowledge and technology exchange between applied researchers and industry.
{"title":"How to Increase the Likelihood of Successful Transfer to Industry -- Going Beyond the Empirical","authors":"T. Gorschek","doi":"10.1109/CESI.2015.9","DOIUrl":"https://doi.org/10.1109/CESI.2015.9","url":null,"abstract":"The field of Empirical Software Engineering has undergone a much-needed expansion the last decade, and papers of all shapes and sizes are more or less mandated to have an \"empirical\" part to be published in premiere venues. The positive trend has researchers realizing the benefits, but also the investments needed, inherent to industry collaboration. That is, real practitioners, involved in the development of software intensive product, system, and service development. This paper shortly summarizes lessons learned from over ten years experience of industrial collaboration, and knowledge and technology exchange between applied researchers and industry.","PeriodicalId":222668,"journal":{"name":"2015 IEEE/ACM 3rd International Workshop on Conducting Empirical Studies in Industry","volume":"191 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2015-05-16","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"116666473","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Conducting empirical research in industry is not an easy task. Previous research has discussed some of the challenges in this type of research and potential solutions have been proposed. However, conducting cross-organizational research introduces specific challenges, some of which are quite hard to overcome. In this paper, we present the use of professional interest groups in a professional social network, Linked In, as a way to recruit participant to an online survey. Using this social network, commonly used among practitioners, was found to be an effective research tool.
{"title":"Finding the Missing Link to Industry: LinkedIn Professional Groups as Facilitators of Empirical Research","authors":"Naomi Unkelos-Shpigel, Sofia Sherman, I. Hadar","doi":"10.1109/CESI.2015.14","DOIUrl":"https://doi.org/10.1109/CESI.2015.14","url":null,"abstract":"Conducting empirical research in industry is not an easy task. Previous research has discussed some of the challenges in this type of research and potential solutions have been proposed. However, conducting cross-organizational research introduces specific challenges, some of which are quite hard to overcome. In this paper, we present the use of professional interest groups in a professional social network, Linked In, as a way to recruit participant to an online survey. Using this social network, commonly used among practitioners, was found to be an effective research tool.","PeriodicalId":222668,"journal":{"name":"2015 IEEE/ACM 3rd International Workshop on Conducting Empirical Studies in Industry","volume":"1 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2015-05-16","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"129034550","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
京公网安备 11010802042870号
京ICP备2023020795号-1
扫码关注我们
求助内容：
应助结果提醒方式：