Pub Date : 2024-07-29DOI: 10.1109/TSE.2024.3435067
Xiaoyan Xu;Filipe R. Cogo;Shane McIntosh
Understanding code coverage is an important precursor to software maintenance activities (e.g., better testing). Although modern code coverage tools provide key insights, they typically rely on code instrumentation, resulting in significant performance overhead. An alternative approach to code instrumentation is to process an application's source code and the associated log traces in tandem. This so-called “log-based code coverage” approach does not impose the same performance overhead as code instrumentation. Chen et al. proposed LogCoCo — a tool that implements log-based code coverage for Java . While LogCoCo breaks important new ground, it has fundamental limitations, namely: uncertainty due to the lack of logging statements in conditional branches, and imprecision caused by dependency injection. In this study, we propose Log2Cov , a tool that generates log-based code coverage for programs written in Python and addresses uncertainty and imprecision issues. We evaluate Log2Cov on three large and active open-source systems. More specifically, we compare the performance of Log2Cov to that of Coverage.py , an instrumentation-based coverage tool for Python . Our results indicate that 1) Log2Cov achieves high precision without introducing runtime overhead; and 2) uncertainty and imprecision can be reduced by up to 11% by statically analyzing the program's source code and execution logs, without requiring additional logging instrumentation from developers. While our enhancements make substantial improvements, we find that future work is needed to handle conditional statements and exception handling blocks to achieve parity with instrumentation-based approaches. We conclude the paper by drawing attention to these promising directions for future work.
{"title":"Mitigating the Uncertainty and Imprecision of Log-Based Code Coverage Without Requiring Additional Logging Statements","authors":"Xiaoyan Xu;Filipe R. Cogo;Shane McIntosh","doi":"10.1109/TSE.2024.3435067","DOIUrl":"10.1109/TSE.2024.3435067","url":null,"abstract":"Understanding code coverage is an important precursor to software maintenance activities (e.g., better testing). Although modern code coverage tools provide key insights, they typically rely on code instrumentation, resulting in significant performance overhead. An alternative approach to code instrumentation is to process an application's source code and the associated log traces in tandem. This so-called “log-based code coverage” approach does not impose the same performance overhead as code instrumentation. Chen et al. proposed \u0000<sc>LogCoCo</small>\u0000 — a tool that implements log-based code coverage for \u0000<sc>Java</small>\u0000. While \u0000<sc>LogCoCo</small>\u0000 breaks important new ground, it has fundamental limitations, namely: uncertainty due to the lack of logging statements in conditional branches, and imprecision caused by dependency injection. In this study, we propose \u0000<sc>Log2Cov</small>\u0000, a tool that generates log-based code coverage for programs written in \u0000<sc>Python</small>\u0000 and addresses uncertainty and imprecision issues. We evaluate \u0000<sc>Log2Cov</small>\u0000 on three large and active open-source systems. More specifically, we compare the performance of \u0000<sc>Log2Cov</small>\u0000 to that of \u0000<sc>Coverage.py</small>\u0000, an instrumentation-based coverage tool for \u0000<sc>Python</small>\u0000. Our results indicate that 1) \u0000<sc>Log2Cov</small>\u0000 achieves high precision without introducing runtime overhead; and 2) uncertainty and imprecision can be reduced by up to 11% by statically analyzing the program's source code and execution logs, without requiring additional logging instrumentation from developers. While our enhancements make substantial improvements, we find that future work is needed to handle conditional statements and exception handling blocks to achieve parity with instrumentation-based approaches. We conclude the paper by drawing attention to these promising directions for future work.","PeriodicalId":13324,"journal":{"name":"IEEE Transactions on Software Engineering","volume":"50 9","pages":"2350-2362"},"PeriodicalIF":6.5,"publicationDate":"2024-07-29","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141794547","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-07-25DOI: 10.1109/TSE.2024.3433463
Jiho Shin;Hadi Hemmati;Moshi Wei;Song Wang
Recently, deep learning models have shown promising results in test oracle generation. Neural Oracle Generation (NOG) models are commonly evaluated using static (automatic) metrics which are mainly based on textual similarity of the output, e.g. BLEU, ROUGE-L, METEOR, and Accuracy. However, these textual similarity metrics may not reflect the testing effectiveness of the generated oracle within a test suite, which is often measured by dynamic (execution-based) test adequacy metrics such as code coverage and mutation score. In this work, we revisit existing oracle generation studies plus gpt-3.5 to empirically investigate the current standing of their performance in textual similarity and test adequacy metrics. Specifically, we train and run four state-of-the-art test oracle generation models on seven textual similarity and two test adequacy metrics for our analysis. We apply two different correlation analyses between these two different sets of metrics. Surprisingly, we found no significant correlation between the textual similarity metrics and test adequacy metrics. For instance, gpt-3.5 on the jackrabbit-oak project had the highest performance on all seven textual similarity metrics among the studied NOGs. However, it had the lowest test adequacy metrics compared to all the studied NOGs. We further conducted a qualitative analysis to explore the reasons behind our observations. We found that oracles with high textual similarity metrics but low test adequacy metrics tend to have complex or multiple chained method invocations within the oracle's parameters, making them hard for the model to generate completely, affecting the test adequacy metrics. On the other hand, oracles with low textual similarity metrics but high test adequacy metrics tend to have to call different assertion types or a different method that functions similarly to the ones in the ground truth. Overall, this work complements prior studies on test oracle generation with an extensive performance evaluation on textual similarity and test adequacy metrics and provides guidelines for better assessment of deep learning applications in software test generation in the future.
最近，深度学习模型在测试甲骨文生成方面取得了可喜的成果。神经甲骨文生成（NOG）模型通常使用静态（自动）指标进行评估，这些指标主要基于输出的文本相似性，如 BLEU、ROUGE-L、METEOR 和 Accuracy。然而，这些文本相似度指标可能无法反映测试套件中生成的甲骨文的测试效果，而测试效果通常是通过动态（基于执行）测试充分性指标（如代码覆盖率和突变分数）来衡量的。在这项工作中，我们重新审视了现有的甲骨文生成研究和 gpt-3.5，以实证研究它们在文本相似性和测试充分性指标方面的性能现状。具体来说，我们在七个文本相似性指标和两个测试充分性指标上训练并运行了四个最先进的测试甲骨文生成模型，以进行分析。我们对这两组不同的指标进行了两种不同的相关性分析。令人惊讶的是，我们发现文本相似度指标和测试充分性指标之间没有明显的相关性。例如，在所研究的 NOG 中，jackrabbit-oak 项目上的 gpt-3.5 在所有七个文本相似度指标上的表现都是最高的。但是，与所有研究的 NOG 相比，它的测试充分性指标最低。我们进一步进行了定性分析，以探索观察结果背后的原因。我们发现，文本相似度指标高但测试充分性指标低的神谕往往在神谕参数中具有复杂或多重链式方法调用，这使得模型难以完全生成，从而影响了测试充分性指标。另一方面，文本相似度指标较低但测试充分性指标较高的神谕往往需要调用不同的断言类型，或调用与基本事实中的断言类型功能类似的不同方法。总之，这项工作通过对文本相似性和测试充分性指标进行广泛的性能评估，对之前关于测试神谕生成的研究进行了补充，并为今后更好地评估深度学习在软件测试生成中的应用提供了指导。
{"title":"Assessing Evaluation Metrics for Neural Test Oracle Generation","authors":"Jiho Shin;Hadi Hemmati;Moshi Wei;Song Wang","doi":"10.1109/TSE.2024.3433463","DOIUrl":"10.1109/TSE.2024.3433463","url":null,"abstract":"Recently, deep learning models have shown promising results in test oracle generation. Neural Oracle Generation (NOG) models are commonly evaluated using static (automatic) metrics which are mainly based on textual similarity of the output, e.g. BLEU, ROUGE-L, METEOR, and Accuracy. However, these textual similarity metrics may not reflect the testing effectiveness of the generated oracle within a test suite, which is often measured by dynamic (execution-based) test adequacy metrics such as code coverage and mutation score. In this work, we revisit existing oracle generation studies plus \u0000<italic>gpt-3.5</i>\u0000 to empirically investigate the current standing of their performance in textual similarity and test adequacy metrics. Specifically, we train and run four state-of-the-art test oracle generation models on seven textual similarity and two test adequacy metrics for our analysis. We apply two different correlation analyses between these two different sets of metrics. Surprisingly, we found no significant correlation between the textual similarity metrics and test adequacy metrics. For instance, \u0000<italic>gpt-3.5</i>\u0000 on the \u0000<italic>jackrabbit-oak</i>\u0000 project had the highest performance on all seven textual similarity metrics among the studied NOGs. However, it had the lowest test adequacy metrics compared to all the studied NOGs. We further conducted a qualitative analysis to explore the reasons behind our observations. We found that oracles with high textual similarity metrics but low test adequacy metrics tend to have complex or multiple chained method invocations within the oracle's parameters, making them hard for the model to generate completely, affecting the test adequacy metrics. On the other hand, oracles with low textual similarity metrics but high test adequacy metrics tend to have to call different assertion types or a different method that functions similarly to the ones in the ground truth. Overall, this work complements prior studies on test oracle generation with an extensive performance evaluation on textual similarity and test adequacy metrics and provides guidelines for better assessment of deep learning applications in software test generation in the future.","PeriodicalId":13324,"journal":{"name":"IEEE Transactions on Software Engineering","volume":"50 9","pages":"2337-2349"},"PeriodicalIF":6.5,"publicationDate":"2024-07-25","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141764147","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-07-22DOI: 10.1109/TSE.2024.3423769
Zimin Chen;Sen Fang;Martin Monperrus
Software optimization refines programs for resource efficiency while preserving functionality. Traditionally, it is a process done by developers and compilers. This paper introduces a third option, automated optimization at the source code level. We present Supersonic , a neural approach targeting minor source code modifications for optimization. Using a seq2seq model, Supersonic is trained on C/C++ program pairs ( $x_{t}$ , $x_{t+1}$ ), where $x_{t+1}$ is an optimized version of $x_{t}$ , and outputs a diff. Supersonic 's performance is benchmarked against OpenAI's GPT-3.5-Turbo and GPT-4 on competitive programming tasks. The experiments show that Supersonic not only outperforms both models on the code optimization task but also minimizes the extent of the change with a model more than 600x smaller than GPT-3.5-Turbo and 3700x smaller than GPT-4.
{"title":"Supersonic: Learning to Generate Source Code Optimizations in C/C++","authors":"Zimin Chen;Sen Fang;Martin Monperrus","doi":"10.1109/TSE.2024.3423769","DOIUrl":"10.1109/TSE.2024.3423769","url":null,"abstract":"Software optimization refines programs for resource efficiency while preserving functionality. Traditionally, it is a process done by developers and compilers. This paper introduces a third option, automated optimization at the source code level. We present \u0000<small>Supersonic</small>\u0000, a neural approach targeting minor source code modifications for optimization. Using a seq2seq model, \u0000<small>Supersonic</small>\u0000 is trained on C/C++ program pairs (\u0000<inline-formula><tex-math>$x_{t}$</tex-math></inline-formula>\u0000, \u0000<inline-formula><tex-math>$x_{t+1}$</tex-math></inline-formula>\u0000), where \u0000<inline-formula><tex-math>$x_{t+1}$</tex-math></inline-formula>\u0000 is an optimized version of \u0000<inline-formula><tex-math>$x_{t}$</tex-math></inline-formula>\u0000, and outputs a diff. \u0000<small>Supersonic</small>\u0000's performance is benchmarked against OpenAI's GPT-3.5-Turbo and GPT-4 on competitive programming tasks. The experiments show that \u0000<small>Supersonic</small>\u0000 not only outperforms both models on the code optimization task but also minimizes the extent of the change with a model more than 600x smaller than GPT-3.5-Turbo and 3700x smaller than GPT-4.","PeriodicalId":13324,"journal":{"name":"IEEE Transactions on Software Engineering","volume":"50 11","pages":"2849-2864"},"PeriodicalIF":6.5,"publicationDate":"2024-07-22","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=10606318","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141754778","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-07-22DOI: 10.1109/TSE.2024.3431585
Qi Mo;Jianeng Wang;Zhongwen Xie;Cong Liu;Fei Dai
Generally, a collaborative business process is a distributed process, in which a set of parallel business processes are involved. These business processes have complementary competencies and knowledge, and cooperate with each other to achieve their common business goals. To ensure the correctness of collaborative business processes, we propose a novel plan-based correctness enforcement approach in this article, which is privacy-preserving, available and efficient. This approach first requires participating organizations to define their business processes. Then, each participating organization employs a set of reduction rules to build the public process of its business process, in which all internal private activities and the flows formed by them are removed. Next, a set of correct plans is generated from these public processes. A plan is essentially a process fragment without alternative routings. From the external perspective (i.e., ignoring all internal private activities and the flows formed by them), a parallel execution of the business processes corresponding to these public processes follows only one such plan. Lastly, each participating organization independently refactors its business process using these resulting correct plans. Using the message places (corresponding to the actual communication interfaces), these refactored processes are composed in parallel. Thus, a correct and loosely coupled enforced process is constructed. This approach is evaluated on actual collaborative business processes, and the experimental results show that compared with state-of-the-art enforcement proposals, it can achieve correctness enforcement while protecting the business privacy of organizations and is available. Meanwhile, it is also more efficient and scalable, even a collaborative business process with tens of millions of states can be enforced within a few seconds.
{"title":"Enforcing Correctness of Collaborative Business Processes Using Plans","authors":"Qi Mo;Jianeng Wang;Zhongwen Xie;Cong Liu;Fei Dai","doi":"10.1109/TSE.2024.3431585","DOIUrl":"10.1109/TSE.2024.3431585","url":null,"abstract":"Generally, a collaborative business process is a distributed process, in which a set of parallel business processes are involved. These business processes have complementary competencies and knowledge, and cooperate with each other to achieve their common business goals. To ensure the correctness of collaborative business processes, we propose a novel plan-based correctness enforcement approach in this article, which is privacy-preserving, available and efficient. This approach first requires participating organizations to define their business processes. Then, each participating organization employs a set of reduction rules to build the public process of its business process, in which all internal private activities and the flows formed by them are removed. Next, a set of correct plans is generated from these public processes. A plan is essentially a process fragment without alternative routings. From the external perspective (i.e., ignoring all internal private activities and the flows formed by them), a parallel execution of the business processes corresponding to these public processes follows only one such plan. Lastly, each participating organization independently refactors its business process using these resulting correct plans. Using the message places (corresponding to the actual communication interfaces), these refactored processes are composed in parallel. Thus, a correct and loosely coupled enforced process is constructed. This approach is evaluated on actual collaborative business processes, and the experimental results show that compared with state-of-the-art enforcement proposals, it can achieve correctness enforcement while protecting the business privacy of organizations and is available. Meanwhile, it is also more efficient and scalable, even a collaborative business process with tens of millions of states can be enforced within a few seconds.","PeriodicalId":13324,"journal":{"name":"IEEE Transactions on Software Engineering","volume":"50 9","pages":"2313-2336"},"PeriodicalIF":6.5,"publicationDate":"2024-07-22","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141754780","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-07-22DOI: 10.1109/TSE.2024.3428972
Sarah Fakhoury;Aaditya Naik;Georgios Sakkas;Saikat Chakraborty;Shuvendu K. Lahiri
Large language models (LLMs) have shown great potential in automating significant aspects of coding by producing natural code from informal natural language (NL) intent. However, given NL is informal, it does not lend easily to checking that the generated code correctly satisfies the user intent. In this paper, we propose a novel interactive workflow TiCoder for guided intent clarification (i.e., partial formalization) through tests to support the generation of more accurate code suggestions. Through a mixed methods user study with 15 programmers, we present an empirical evaluation of the effectiveness of the workflow to improve code generation accuracy. We find that participants using the proposed workflow are significantly more likely to correctly evaluate AI generated code, and report significantly less task-induced cognitive load. Furthermore, we test the potential of the workflow at scale with four different state-of-the-art LLMs on two python datasets, using an idealized proxy for a user feedback. We observe an average absolute improvement of 45.97% in the pass@1 code generation accuracy for both datasets and across all LLMs within 5 user interactions, in addition to the automatic generation of accompanying unit tests.
{"title":"LLM-Based Test-Driven Interactive Code Generation: User Study and Empirical Evaluation","authors":"Sarah Fakhoury;Aaditya Naik;Georgios Sakkas;Saikat Chakraborty;Shuvendu K. Lahiri","doi":"10.1109/TSE.2024.3428972","DOIUrl":"10.1109/TSE.2024.3428972","url":null,"abstract":"Large language models (LLMs) have shown great potential in automating significant aspects of coding by producing natural code from informal natural language (NL) intent. However, given NL is informal, it does not lend easily to checking that the generated code correctly satisfies the user intent. In this paper, we propose a novel interactive workflow \u0000<sc>TiCoder</small>\u0000 for guided intent clarification (i.e., partial formalization) through tests to support the generation of more accurate code suggestions. Through a mixed methods user study with 15 programmers, we present an empirical evaluation of the effectiveness of the workflow to improve code generation accuracy. We find that participants using the proposed workflow are significantly more likely to correctly evaluate AI generated code, and report significantly less task-induced cognitive load. Furthermore, we test the potential of the workflow at scale with four different state-of-the-art LLMs on two python datasets, using an idealized proxy for a user feedback. We observe an average absolute improvement of 45.97% in the pass@1 code generation accuracy for both datasets and across all LLMs within 5 user interactions, in addition to the automatic generation of accompanying unit tests.","PeriodicalId":13324,"journal":{"name":"IEEE Transactions on Software Engineering","volume":"50 9","pages":"2254-2268"},"PeriodicalIF":6.5,"publicationDate":"2024-07-22","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141754777","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-07-22DOI: 10.1109/TSE.2024.3431445
Zhe Yu;Joymallya Chakraborty;Tim Menzies
This research seeks to benefit the software engineering society by providing a simple yet effective pre-processing approach to achieve equalized odds fairness in machine learning software. Fairness issues have attracted increasing attention since machine learning software is increasingly used for high-stakes and high-risk decisions. It is the responsibility of all software developers to make their software accountable by ensuring that the machine learning software do not perform differently on different sensitive demographic groups—satisfying equalized odds. Different from prior works which either optimize for an equalized odds related metric during the learning process like a black-box, or manipulate the training data following some intuition; this work studies the root cause of the violation of equalized odds and how to tackle it. We found that equalizing the class distribution in each demographic group with sample weights is a necessary condition for achieving equalized odds without modifying the normal training process. In addition, an important partial condition for equalized odds (zero average odds difference) can be guaranteed when the class distributions are weighted to be not only equal but also balanced (1:1). Based on these analyses, we proposed FairBalance, a pre-processing algorithm which balances the class distribution in each demographic group by assigning calculated weights to the training data. On eight real-world datasets, our empirical results show that, at low computational overhead, the proposed pre-processing algorithm FairBalance can significantly improve equalized odds without much, if any damage to the utility. FairBalance also outperforms existing state-of-the-art approaches in terms of equalized odds. To facilitate reuse, reproduction, and validation, we made our scripts available at https://github.com/hil-se/FairBalance .
{"title":"FairBalance: How to Achieve Equalized Odds With Data Pre-Processing","authors":"Zhe Yu;Joymallya Chakraborty;Tim Menzies","doi":"10.1109/TSE.2024.3431445","DOIUrl":"10.1109/TSE.2024.3431445","url":null,"abstract":"This research seeks to benefit the software engineering society by providing a simple yet effective pre-processing approach to achieve equalized odds fairness in machine learning software. Fairness issues have attracted increasing attention since machine learning software is increasingly used for high-stakes and high-risk decisions. It is the responsibility of all software developers to make their software accountable by ensuring that the machine learning software do not perform differently on different sensitive demographic groups—satisfying equalized odds. Different from prior works which either optimize for an equalized odds related metric during the learning process like a black-box, or manipulate the training data following some intuition; this work studies the root cause of the violation of equalized odds and how to tackle it. We found that equalizing the class distribution in each demographic group with sample weights is a necessary condition for achieving equalized odds without modifying the normal training process. In addition, an important partial condition for equalized odds (zero average odds difference) can be guaranteed when the class distributions are weighted to be not only equal but also balanced (1:1). Based on these analyses, we proposed FairBalance, a pre-processing algorithm which balances the class distribution in each demographic group by assigning calculated weights to the training data. On eight real-world datasets, our empirical results show that, at low computational overhead, the proposed pre-processing algorithm FairBalance can significantly improve equalized odds without much, if any damage to the utility. FairBalance also outperforms existing state-of-the-art approaches in terms of equalized odds. To facilitate reuse, reproduction, and validation, we made our scripts available at \u0000<uri>https://github.com/hil-se/FairBalance</uri>\u0000.","PeriodicalId":13324,"journal":{"name":"IEEE Transactions on Software Engineering","volume":"50 9","pages":"2294-2312"},"PeriodicalIF":6.5,"publicationDate":"2024-07-22","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141754779","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-07-18DOI: 10.1109/TSE.2024.3430514
Ana C. Marcén;Antonio Iglesias;Raúl Lapeña;Francisca Pérez;Carlos Cetina
Model-driven engineering (MDE) is a software engineering paradigm based on the systematic use of models. Over the past few years, engineers have significantly increased the use of MDE, which has been reported as a successful paradigm for developing industrial software. Recently, there have also been remarkable advancements in the Artificial Intelligence (AI) domain, with a significant increase in advanced Machine Learning (ML) techniques. The advances in both fields have led to a surge in works that dwell within the intersection of ML and MDE. This work places the focus on systematically reviewing works that leverage ML to solve MDE problems. We have reviewed a total of 9,194 papers, selecting 98 studies for further analysis. The results of our Systematic Literature Review (SLR) bring light to the current state of the art and trends in the field, discussing the drift in the usage of the different available ML techniques along with the remaining research gaps and open challenges. Our SLR has the potential to produce a positive impact in the research community by steering it towards ML techniques that have been successfully applied to solve MDE challenges.
模型驱动工程（MDE）是一种基于系统使用模型的软件工程范式。在过去几年中，工程师们大幅增加了对 MDE 的使用，据报道，MDE 已成为开发工业软件的成功范例。最近，人工智能（AI）领域也取得了显著进步，先进的机器学习（ML）技术大幅增加。这两个领域的进步导致了 ML 和 MDE 交叉领域作品的激增。这项工作的重点是系统回顾利用 ML 解决 MDE 问题的作品。我们总共查阅了 9,194 篇论文，从中挑选出 98 项研究进行进一步分析。我们的系统性文献综述（SLR）结果揭示了该领域的技术现状和发展趋势，讨论了不同可用 ML 技术的使用偏移以及剩余的研究空白和公开挑战。我们的系统文献综述有可能对研究界产生积极影响，引导他们转向已成功应用于解决 MDE 挑战的 ML 技术。
{"title":"A Systematic Literature Review of Model-Driven Engineering Using Machine Learning","authors":"Ana C. Marcén;Antonio Iglesias;Raúl Lapeña;Francisca Pérez;Carlos Cetina","doi":"10.1109/TSE.2024.3430514","DOIUrl":"10.1109/TSE.2024.3430514","url":null,"abstract":"Model-driven engineering (MDE) is a software engineering paradigm based on the systematic use of models. Over the past few years, engineers have significantly increased the use of MDE, which has been reported as a successful paradigm for developing industrial software. Recently, there have also been remarkable advancements in the Artificial Intelligence (AI) domain, with a significant increase in advanced Machine Learning (ML) techniques. The advances in both fields have led to a surge in works that dwell within the intersection of ML and MDE. This work places the focus on systematically reviewing works that leverage ML to solve MDE problems. We have reviewed a total of 9,194 papers, selecting 98 studies for further analysis. The results of our Systematic Literature Review (SLR) bring light to the current state of the art and trends in the field, discussing the drift in the usage of the different available ML techniques along with the remaining research gaps and open challenges. Our SLR has the potential to produce a positive impact in the research community by steering it towards ML techniques that have been successfully applied to solve MDE challenges.","PeriodicalId":13324,"journal":{"name":"IEEE Transactions on Software Engineering","volume":"50 9","pages":"2269-2293"},"PeriodicalIF":6.5,"publicationDate":"2024-07-18","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141725777","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-07-15DOI: 10.1109/TSE.2024.3428324
Mona Nashaat;James Miller
Large language models like BERT and GPT possess significant capabilities and potential impacts across various applications. Software engineers often use these models for code-related tasks, including generating, debugging, and summarizing code. Nevertheless, large language models still have several flaws, including model hallucination. (e.g., generating erroneous code and producing outdated and inaccurate programs) and the substantial computational resources and energy required for training and fine-tuning. To tackle these challenges, we propose CodeMentor, a framework for few-shot learning to train large language models with the data available within the organization. We employ the framework to train a language model for code review activities, such as code refinement and review generation. The framework utilizes heuristic rules and weak supervision techniques to leverage available data, such as previous review comments, issue reports, and related code updates. Then, the framework employs the constructed dataset to fine-tune LLMs for code review tasks. Additionally, the framework integrates domain expertise by employing reinforcement learning with human feedback. This allows domain experts to assess the generated code and enhance the model performance. Also, to assess the performance of the proposed model, we evaluate it with four state-of-the-art techniques in various code review tasks. The experimental results attest that CodeMentor enhances the performance in all tasks compared to the state-of-the-art approaches, with an improvement of up to 22.3%, 43.4%, and 24.3% in code quality estimation, review generation, and bug report summarization tasks, respectively.
{"title":"Towards Efficient Fine-Tuning of Language Models With Organizational Data for Automated Software Review","authors":"Mona Nashaat;James Miller","doi":"10.1109/TSE.2024.3428324","DOIUrl":"10.1109/TSE.2024.3428324","url":null,"abstract":"Large language models like BERT and GPT possess significant capabilities and potential impacts across various applications. Software engineers often use these models for code-related tasks, including generating, debugging, and summarizing code. Nevertheless, large language models still have several flaws, including model hallucination. (e.g., generating erroneous code and producing outdated and inaccurate programs) and the substantial computational resources and energy required for training and fine-tuning. To tackle these challenges, we propose CodeMentor, a framework for few-shot learning to train large language models with the data available within the organization. We employ the framework to train a language model for code review activities, such as code refinement and review generation. The framework utilizes heuristic rules and weak supervision techniques to leverage available data, such as previous review comments, issue reports, and related code updates. Then, the framework employs the constructed dataset to fine-tune LLMs for code review tasks. Additionally, the framework integrates domain expertise by employing reinforcement learning with human feedback. This allows domain experts to assess the generated code and enhance the model performance. Also, to assess the performance of the proposed model, we evaluate it with four state-of-the-art techniques in various code review tasks. The experimental results attest that CodeMentor enhances the performance in all tasks compared to the state-of-the-art approaches, with an improvement of up to 22.3%, 43.4%, and 24.3% in code quality estimation, review generation, and bug report summarization tasks, respectively.","PeriodicalId":13324,"journal":{"name":"IEEE Transactions on Software Engineering","volume":"50 9","pages":"2240-2253"},"PeriodicalIF":6.5,"publicationDate":"2024-07-15","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141625007","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
To improve the quality of Android apps, developers use automated debugging and testing solutions to determine whether the previously found crashes are reproducible. However, existing GUI fuzzing solutions for Android apps struggle to reproduce crashes efficiently based solely on a crash stack trace. This trace provides the location in the app where the crash occurs. GUI fuzzing solutions currently in use rely on heuristics to generate UI events. Unfortunately, these events often do not align with the investigation of an app's UI event space to reach a specific location of code. Hence, they generate numerous events unrelated to the crash, leading to an event explosion. To address this issue, a precise static UI model of widgets and screens can greatly enhance the efficiency of a fuzzing tool in its search. Building such a model requires considering all possible combinations of event sequences on widgets since the execution order of events is not statically determined. However, this approach presents scalability challenges in complex apps with several widgets. In this paper, we propose a directed-based fuzzing solution to reduce an app's event domain to the necessary ones to trigger a crash. Our insight is that the dependencies between widgets in their visual presentation and attribute states provide valuable information in precisely identifying events that trigger a crash. We propose an attribute-sensitive reachability analysis (ASRA) to track dependent widgets in reachable paths to the crash point and distinguish between events in terms of their relevancy to be generated in the crash reproduction process. With instrumentation, we inject code to prune irrelevant events, reducing the event domain to search at run time. We used four famous fuzzing tools, Monkey, Ape, Stoat, and FastBot2, to assess the impact of our solution in decreasing the crash reproduction time and increasing the possibility of reproducing a crash. Our results show that the success ratio of reproducing a crash has increased for one-fourth of crashes. In addition, the average reproduction time of a crash becomes at least 2x faster. Wilcoxon Mann-Whitney test shows this enhancement is significant when our tool is used compared to baseline and insensitive reachability analysis.
{"title":"Mole: Efficient Crash Reproduction in Android Applications With Enforcing Necessary UI Events","authors":"Maryam Masoudian;Heqing Huang;Morteza Amini;Charles Zhang","doi":"10.1109/TSE.2024.3428543","DOIUrl":"10.1109/TSE.2024.3428543","url":null,"abstract":"To improve the quality of Android apps, developers use automated debugging and testing solutions to determine whether the previously found crashes are reproducible. However, existing GUI fuzzing solutions for Android apps struggle to reproduce crashes efficiently based solely on a crash stack trace. This trace provides the location in the app where the crash occurs. GUI fuzzing solutions currently in use rely on heuristics to generate UI events. Unfortunately, these events often do not align with the investigation of an app's UI event space to reach a specific location of code. Hence, they generate numerous events unrelated to the crash, leading to an event explosion. To address this issue, a precise static UI model of widgets and screens can greatly enhance the efficiency of a fuzzing tool in its search. Building such a model requires considering all possible combinations of event sequences on widgets since the execution order of events is not statically determined. However, this approach presents scalability challenges in complex apps with several widgets. In this paper, we propose a directed-based fuzzing solution to reduce an app's event domain to the necessary ones to trigger a crash. Our insight is that the dependencies between widgets in their visual presentation and attribute states provide valuable information in precisely identifying events that trigger a crash. We propose an attribute-sensitive reachability analysis (ASRA) to track dependent widgets in reachable paths to the crash point and distinguish between events in terms of their relevancy to be generated in the crash reproduction process. With instrumentation, we inject code to prune irrelevant events, reducing the event domain to search at run time. We used four famous fuzzing tools, Monkey, Ape, Stoat, and FastBot2, to assess the impact of our solution in decreasing the crash reproduction time and increasing the possibility of reproducing a crash. Our results show that the success ratio of reproducing a crash has increased for \u0000<italic>one-fourth</i>\u0000 of crashes. In addition, the average reproduction time of a crash becomes at least 2x faster. Wilcoxon Mann-Whitney test shows this enhancement is significant when our tool is used compared to baseline and insensitive reachability analysis.","PeriodicalId":13324,"journal":{"name":"IEEE Transactions on Software Engineering","volume":"50 8","pages":"2200-2218"},"PeriodicalIF":6.5,"publicationDate":"2024-07-15","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141625080","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
京公网安备 11010802042870号
京ICP备2023020795号-1
扫码关注我们
求助内容：
应助结果提醒方式：