Information and Software Technology最新文献

{"title":"Qubernetes: Towards a unified cloud-native execution platform for hybrid classic-quantum computing","authors":"Vlad Stirbu,&nbsp;Otso Kinanen,&nbsp;Majid Haghparast,&nbsp;Tommi Mikkonen","doi":"10.1016/j.infsof.2024.107529","DOIUrl":"10.1016/j.infsof.2024.107529","url":null,"abstract":"<div><h3>Context:</h3><p>The emergence of quantum computing proposes a revolutionary paradigm that can radically transform numerous scientific and industrial application domains. The ability of quantum computers to scale computations beyond what the current computers are capable of implies better performance and efficiency for certain algorithmic tasks.</p></div><div><h3>Objective:</h3><p>However, to benefit from such improvement, quantum computers must be integrated with existing software systems, a process that is not straightforward. In this paper, we propose a unified execution model that addresses the challenges that emerge from building hybrid classical-quantum applications at scale.</p></div><div><h3>Method:</h3><p>Following the Design Science Research methodology, we proposed a convention for mapping quantum resources and artifacts to Kubernetes concepts. Then, in an experimental Kubernetes cluster, we conducted experiments for scheduling and executing quantum tasks on both quantum simulators and hardware.</p></div><div><h3>Results:</h3><p>The experimental results demonstrate that the proposed platform Qubernetes (or Kubernetes for quantum) exposes the quantum computation tasks and hardware capabilities following established cloud-native principles, allowing seamless integration into the larger Kubernetes ecosystem.</p></div><div><h3>Conclusion:</h3><p>The quantum computing potential cannot be realized without seamless integration into classical computing. By validating that it is practical to execute quantum tasks in a Kubernetes infrastructure, we pave the way for leveraging the existing Kubernetes ecosystem as an enabler for hybrid classical-quantum computing.</p></div>","PeriodicalId":54983,"journal":{"name":"Information and Software Technology","volume":"175 ","pages":"Article 107529"},"PeriodicalIF":3.8,"publicationDate":"2024-07-26","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.sciencedirect.com/science/article/pii/S0950584924001344/pdfft?md5=f5a427a08f3dbd8b3b7ccb4f62b577ea&pid=1-s2.0-S0950584924001344-main.pdf","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141838848","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Extraction and empirical evaluation of GUI-level invariants as GUI Oracles in mobile app testing","authors":"Ali Asghar Yarifard ,&nbsp;Saeed Araban ,&nbsp;Samad Paydar ,&nbsp;Vahid Garousi ,&nbsp;Maurizio Morisio ,&nbsp;Riccardo Coppola","doi":"10.1016/j.infsof.2024.107531","DOIUrl":"10.1016/j.infsof.2024.107531","url":null,"abstract":"<div><h3>Context</h3><p>Mobile apps (software) are used in almost all aspects of daily life by billions of people. Given the widespread use of mobile apps in various domains, the demand for systematic testing of their Graphical User Interfaces (GUI) is crucial. Despite the significant advances in automated mobile app testing over the last decade, certain challenges remain, most notably the app-specific GUI test-oracle problem, which can significantly hinder the effective detection of defects in mobile apps. In this study, we introduce the use of GUI-level invariants, referred to as GUI invariants, as app-specific GUI oracles in GUI test cases to address this challenge.</p></div><div><h3>Methods</h3><p>We propose a semi-automatic solution to extract GUI invariants and use them as app-specific GUI oracles in test cases. We use the mutation testing technique to evaluate the (fault detection) effectiveness of the GUI oracles used. In addition, we evaluate their quality aspects, namely correctness, understandability, and compatibility, from the perspective of human experts using a questionnaire survey.</p></div><div><h3>Results</h3><p>The empirical results show that the GUI oracles used are effective and helpful, as they improved the fault-detection effectiveness of the empirical test suites ranging from 18% to 32%. These results also highlight the efficacy of GUI oracles used in identifying various defects, including crashing and non-crashing functional issues, and surpassing the performance of existing tools in fault-detection rates. Additionally, the questionnaire survey outcomes indicate that the GUI oracles used are correct, understandable, and compatible.</p></div><div><h3>Conclusions</h3><p>Based on the empirical results, we can conclude that using GUI invariants as GUI oracles can be useful and effective in mobile app testing.</p></div>","PeriodicalId":54983,"journal":{"name":"Information and Software Technology","volume":"177 ","pages":"Article 107531"},"PeriodicalIF":3.8,"publicationDate":"2024-07-23","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.sciencedirect.com/science/article/pii/S0950584924001368/pdfft?md5=df482f7792ae43af274444769943b80c&pid=1-s2.0-S0950584924001368-main.pdf","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141844371","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"A PRISMA-driven systematic mapping study on system assurance weakeners","authors":"Kimya Khakzad Shahandashti ,&nbsp;Alvine B. Belle ,&nbsp;Timothy C. Lethbridge ,&nbsp;Oluwafemi Odu ,&nbsp;Mithila Sivakumar","doi":"10.1016/j.infsof.2024.107526","DOIUrl":"10.1016/j.infsof.2024.107526","url":null,"abstract":"<div><h3>Context:</h3><p>An assurance case is a structured hierarchy of claims aiming at demonstrating that a mission-critical system supports specific requirements (e.g., safety, security, privacy). The presence of assurance weakeners (i.e., assurance deficits, logical fallacies) in assurance cases reflects insufficient evidence, knowledge, or gaps in reasoning. These weakeners can undermine confidence in assurance arguments, potentially hindering the verification of mission-critical system capabilities which could result in catastrophic outcomes (e.g., loss of lives). Given the growing interest in employing assurance cases to ensure that systems are developed to meet their requirements, exploring the management of assurance weakeners becomes beneficial.</p></div><div><h3>Objective:</h3><p>As a stepping stone for future research on assurance weakeners, we aim to initiate the first comprehensive systematic mapping study on this subject.</p></div><div><h3>Methods:</h3><p>We followed the well-established PRISMA 2020 and SEGRESS guidelines to conduct our systematic mapping study. We searched for primary studies in five digital libraries and focused on the 2012–2023 publication year range. Our selection criteria focused on studies addressing assurance weakeners from a qualitative standpoint, resulting in the inclusion of 39 primary studies in our systematic review.</p></div><div><h3>Results:</h3><p>Our systematic mapping study reports a taxonomy (map) that provides a uniform categorization of assurance weakeners and approaches proposed to manage them from a qualitative perspective. The taxonomy classifies weakeners in four categories: aleatory, epistemic, ontological, and argument uncertainty. Additionally, it classifies approaches supporting the management of weakeners in three main categories: representation, identification and mitigation approaches.</p></div><div><h3>Conclusion:</h3><p>Our study findings suggest that the SACM (Structured Assurance Case Metamodel) – a standard specified by the OMG (Object Management Group) – offers a comprehensive range of capabilities to capture structured arguments and reason about their potential assurance weakeners. Our findings also suggest novel assurance weakener management approaches should be proposed to better assure mission-critical systems.</p></div>","PeriodicalId":54983,"journal":{"name":"Information and Software Technology","volume":"175 ","pages":"Article 107526"},"PeriodicalIF":3.8,"publicationDate":"2024-07-20","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.sciencedirect.com/science/article/pii/S0950584924001319/pdfft?md5=5f5782fecf500fafd6a0caffa9ef549f&pid=1-s2.0-S0950584924001319-main.pdf","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141846295","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"An empirical study on compatibility issues in Android API field evolution","authors":"Tarek Mahmud ,&nbsp;Meiru Che ,&nbsp;Guowei Yang","doi":"10.1016/j.infsof.2024.107530","DOIUrl":"10.1016/j.infsof.2024.107530","url":null,"abstract":"<div><h3>Context:</h3><p>The continuous evolution of the Android operating system requires regular API updates, which may affect the functionality of Android apps. This is becoming increasingly common due to the frequent evolution of the Android platform, which introduces new APIs and deprecates existing ones. Recent studies investigated API evolution to ensure the reliability of Android apps; however, they focused on API methods alone.</p></div><div><h3>Objectives:</h3><p>This study aims to understand how API fields evolve and how this affects API compatibility in real-world Android apps and their development.</p></div><div><h3>Method:</h3><p>We perform an empirical study on compatibility issues in Android API field evolution by analyzing the nature and resolution of these issues across 681 open-source Android apps.</p></div><div><h3>Results:</h3><p>Our experimental results yield interesting findings: (1) On average two API field compatibility issues exist per app in each tag; (2) Although API method evolution and API field evolution are related, current API method-level analysis techniques may fail to detect numerous API field compatibility issues; (3) Different types of checks are preferred when addressing different types of compatibility issues; (4) It takes on average three and a half months for an API field compatibility issue to get fixed since when it is introduced; (5) Developers pay proper attention to API field compatibility issues and address them soon after becoming aware of them in the apps.</p></div><div><h3>Conclusion:</h3><p>These findings highlight the significance of including API fields in future research on API evolution and can assist developers and researchers in understanding, detecting, and handling compatibility issues in API field evolution.</p></div>","PeriodicalId":54983,"journal":{"name":"Information and Software Technology","volume":"175 ","pages":"Article 107530"},"PeriodicalIF":3.8,"publicationDate":"2024-07-20","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141846783","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"The quantum frontier of software engineering: A systematic mapping study","authors":"Manuel De Stefano,&nbsp;Fabiano Pecorelli,&nbsp;Dario Di Nucci,&nbsp;Fabio Palomba,&nbsp;Andrea De Lucia","doi":"10.1016/j.infsof.2024.107525","DOIUrl":"10.1016/j.infsof.2024.107525","url":null,"abstract":"<div><h3>Context:</h3><p>Quantum computing is becoming a reality, and quantum software engineering (QSE) is emerging as a new discipline to enable developers to design and develop quantum programs.</p></div><div><h3>Objective:</h3><p>This paper presents a systematic mapping study of the current state of QSE research, aiming to identify the most investigated topics, the types and number of studies, the main reported results, and the most studied quantum computing tools/frameworks. Additionally, the study aims to explore the research community’s interest in QSE, how it has evolved, and any prior contributions to the discipline before its formal introduction through the Talavera Manifesto.</p></div><div><h3>Method:</h3><p>We searched for relevant articles in several databases and applied inclusion and exclusion criteria to select the most relevant studies. After evaluating the quality of the selected resources, we extracted relevant data from the primary studies and analyzed them.</p></div><div><h3>Results:</h3><p>We found that QSE research has primarily focused on software testing, with little attention given to other topics, such as software engineering management. The most commonly studied technology for techniques and tools is Qiskit, although, in most studies, either multiple or none specific technologies were employed. The researchers most interested in QSE are interconnected through direct collaborations, and several strong collaboration clusters have been identified. Most articles in QSE have been published in non-thematic venues, with a preference for conferences.</p></div><div><h3>Conclusions:</h3><p>The study’s implications are providing a centralized source of information for researchers and practitioners in the field, facilitating knowledge transfer, and contributing to the advancement and growth of QSE.</p></div>","PeriodicalId":54983,"journal":{"name":"Information and Software Technology","volume":"175 ","pages":"Article 107525"},"PeriodicalIF":3.8,"publicationDate":"2024-07-18","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.sciencedirect.com/science/article/pii/S0950584924001307/pdfft?md5=d0ccf8db440c2e6dfde8fed9283c96a6&pid=1-s2.0-S0950584924001307-main.pdf","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141933145","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"An exploratory study on just-in-time multi-programming-language bug prediction","authors":"Zengyang Li ,&nbsp;Jiabao Ji ,&nbsp;Peng Liang ,&nbsp;Ran Mo ,&nbsp;Hui Liu","doi":"10.1016/j.infsof.2024.107524","DOIUrl":"10.1016/j.infsof.2024.107524","url":null,"abstract":"<div><h3>Context:</h3><p>An increasing number of software systems are written in multiple programming languages (PLs), which are called multi-programming-language (MPL) systems. MPL bugs (MPLBs) refers to the bugs whose resolution involves multiple PLs. Despite high complexity of MPLB resolution, there lacks MPLB prediction methods.</p></div><div><h3>Objective:</h3><p>This work aims to construct just-in-time (JIT) MPLB prediction models with selected prediction metrics, analyze the significance of the metrics, and then evaluate the performance of cross-project JIT MPLB prediction.</p></div><div><h3>Methods:</h3><p>We develop JIT MPLB prediction models with the selected metrics using machine learning algorithms and evaluate the models in within-project and cross-project contexts with our constructed dataset based on 18 Apache MPL projects.</p></div><div><h3>Results:</h3><p>Random Forest is appropriate for JIT MPLB prediction. Changed LOC of all files, added LOC of all files, and the total number of lines of all files of the project currently are the most crucial metrics in JIT MPLB prediction. The prediction models can be simplified using a few top-ranked metrics. Training on the dataset from multiple projects can yield significantly higherAUC than training on the dataset from a single project for cross-project JIT MPLB prediction.</p></div><div><h3>Conclusions:</h3><p>JIT MPLB prediction models can be constructed with the selected set of metrics, which can be reduced to build simplified JIT MPLB prediction models, and cross-project JIT MPLB prediction is feasible.</p></div>","PeriodicalId":54983,"journal":{"name":"Information and Software Technology","volume":"175 ","pages":"Article 107524"},"PeriodicalIF":3.8,"publicationDate":"2024-07-18","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141853189","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"SFIDMT-ART: A metamorphic group generation method based on Adaptive Random Testing applied to source and follow-up input domains","authors":"Zhihao Ying ,&nbsp;Dave Towey ,&nbsp;Anthony Graham Bellotti ,&nbsp;Tsong Yueh Chen ,&nbsp;Zhi Quan Zhou","doi":"10.1016/j.infsof.2024.107528","DOIUrl":"10.1016/j.infsof.2024.107528","url":null,"abstract":"<div><h3>Context:</h3><p>The performance of metamorphic testing relates strongly to the quality of test cases. However, most related research has only focused on source test cases, ignoring follow-up test cases to some extent. In this paper, we identify a potential problem that may be encountered with existing metamorphic group generation algorithms. We then propose a possible solution to address this problem. Based on this solution, we design a new algorithm for generating effective source and follow-up test cases.</p></div><div><h3>Objective:</h3><p>To improve the performance (test effectiveness and efficiency) of metamorphic testing.</p></div><div><h3>Methods:</h3><p>We introduce the concept of the input-domain difference problem, which is likely to affect the performance of metamorphic group generation algorithms. We propose a new test-case distribution criterion for metamorphic testing to address this problem. Based on our proposed criterion, we further present a new metamorphic group generation algorithm, from a black-box perspective, with new distance metrics to facilitate this algorithm.</p></div><div><h3>Results:</h3><p>Our algorithm performs significantly better than existing algorithms, in terms of test effectiveness, efficiency and test-case diversity.</p></div><div><h3>Conclusions:</h3><p>Through experiments, we find that the input-domain difference problem is likely to affect the performance of metamorphic group generation algorithms. The experimental results demonstrate that our algorithm can achieve good test efficiency, effectiveness, and test-case diversity.</p></div>","PeriodicalId":54983,"journal":{"name":"Information and Software Technology","volume":"175 ","pages":"Article 107528"},"PeriodicalIF":3.8,"publicationDate":"2024-07-18","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.sciencedirect.com/science/article/pii/S0950584924001332/pdfft?md5=47d3eca48ca9d0d5eb3854e033268e84&pid=1-s2.0-S0950584924001332-main.pdf","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141844581","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Cross-Modal Retrieval-enhanced code Summarization based on joint learning for retrieval and generation","authors":"Lixuan Li ,&nbsp;Bin Liang ,&nbsp;Lin Chen ,&nbsp;Xiaofang Zhang","doi":"10.1016/j.infsof.2024.107527","DOIUrl":"10.1016/j.infsof.2024.107527","url":null,"abstract":"<div><h3>Context:</h3><p>Code summarization refers to a task that automatically generates a natural language description of a code snippet to facilitate code comprehension. Existing methods have achieved satisfactory results by incorporating information retrieval into generative deep-learning models for reusing summaries of existing code. However, most of these existing methods employed non-learnable generic retrieval methods for content-based retrieval, resulting in a lack of diversity in the retrieved results during training, thereby making the model over-reliant on retrieved results and reducing the generative model’s ability to generalize to unknown samples.</p></div><div><h3>Objective:</h3><p>To address this issue, this paper introduces CMR-Sum: a novel Cross-Modal Retrieval-enhanced code Summarization framework based on joint learning for generation and retrieval tasks, where both two tasks are allowed to be optimized simultaneously.</p></div><div><h3>Method:</h3><p>Specifically, we use a cross-modal retrieval module to dynamically alter retrieval results during training, which enhances the diversity of the retrieved results and maintains a relative balance between the two tasks. Furthermore, in the summary generation phase, we employ a cross-attention mechanism to generate code summaries based on the alignment between retrieved and generated summaries. We conducted experiments on three real-world datasets, comparing the performance of our method with baseline models. Additionally, we performed extensive qualitative analysis.</p></div><div><h3>Result:</h3><p>Results from qualitative and quantitative experiments indicate that our approach effectively enhances the performance of code summarization. Our method outperforms both the generation-based and the retrieval-enhanced baselines. Further ablation experiments demonstrate the effectiveness of each component of our method. Results from sensitivity analysis experiments suggest that our approach achieves good performance without requiring extensive hyper-parameter search.</p></div><div><h3>Conclusion:</h3><p>The direction of utilizing retrieval-enhanced generation tasks shows great potential. It is essential to increase the diversity of retrieval results during the training process, which is crucial for improving the generality and the performance of the model.</p></div>","PeriodicalId":54983,"journal":{"name":"Information and Software Technology","volume":"175 ","pages":"Article 107527"},"PeriodicalIF":3.8,"publicationDate":"2024-07-18","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141851202","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"GitHub marketplace for automation and innovation in software production","authors":"SK. Golam Saroar,&nbsp;Waseefa Ahmed,&nbsp;Elmira Onagh,&nbsp;Maleknaz Nayebi","doi":"10.1016/j.infsof.2024.107522","DOIUrl":"10.1016/j.infsof.2024.107522","url":null,"abstract":"<div><h3>Context:</h3><p>GitHub, renowned for facilitating collaborative code version control and software production in software teams, expanded its services in 2017 by introducing GitHub Marketplace. This online platform hosts automation tools to assist developers with the production of their GitHub-hosted projects, and it has become a valuable source of information on the tools used in the Open Source Software (OSS) community.</p></div><div><h3>Objective:</h3><p>In this exploratory study, we introduce GitHub Marketplace as a software marketplace by exploring the Characteristics, Features, and Policies of the platform comprehensively, identifying common themes in production automation. Further, we explore popular tools among practitioners and researchers and highlight disparities in the approach to these tools between industry and academia.</p></div><div><h3>Method:</h3><p>We adopted the conceptual framework of software app stores from previous studies and used that to examine 8,318 automated production tools (440 Apps and 7,878 Actions) across 32 categories on GitHub Marketplace. We explored and described the policies of this marketplace as a unique platform where developers share production tools for the use of other developers. Furthermore, we conducted a systematic mapping of 515 research papers published from 2000 to 2021 and compared open-source academic production tools with those available in the marketplace.</p></div><div><h3>Results:</h3><p>We found that although some of the automation topics in literature are widely used in practice, they have yet to align with the state-of-practice for automated production. We discovered that practitioners often use automation tools for tasks like “Continuous Integration” and “Utilities”, while researchers tend to focus more on “Code Quality” and “Testing”.</p></div><div><h3>Conclusion:</h3><p>Our study illuminates the landscape of open-source tools for automation production. We also explored the disparities between industry trends and researchers’ priorities. Recognizing these distinctions can empower researchers to build on existing work and guide practitioners in selecting tools that meet their specific needs. Bridging this gap between industry and academia helps with further innovation in the field and ensures that research remains pertinent to the evolving challenges in software production.</p></div>","PeriodicalId":54983,"journal":{"name":"Information and Software Technology","volume":"175 ","pages":"Article 107522"},"PeriodicalIF":3.8,"publicationDate":"2024-07-14","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.sciencedirect.com/science/article/pii/S0950584924001277/pdfft?md5=a5f34bd55e7fac78519b3f67fd64a5c6&pid=1-s2.0-S0950584924001277-main.pdf","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141693361","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Fine-tuning and prompt engineering for large language models-based code review automation","authors":"Chanathip Pornprasit,&nbsp;Chakkrit Tantithamthavorn","doi":"10.1016/j.infsof.2024.107523","DOIUrl":"10.1016/j.infsof.2024.107523","url":null,"abstract":"<div><h3>Context:</h3><p>The rapid evolution of Large Language Models (LLMs) has sparked significant interest in leveraging their capabilities for automating code review processes. Prior studies often focus on developing LLMs for code review automation, yet require expensive resources, which is infeasible for organizations with limited budgets and resources. Thus, fine-tuning and prompt engineering are the two common approaches to leveraging LLMs for code review automation.</p></div><div><h3>Objective:</h3><p>We aim to investigate the performance of LLMs-based code review automation based on two contexts, i.e., when LLMs are leveraged by fine-tuning and prompting. Fine-tuning involves training the model on a specific code review dataset, while prompting involves providing explicit instructions to guide the model’s generation process without requiring a specific code review dataset.</p></div><div><h3>Methods:</h3><p>We leverage model fine-tuning and inference techniques (i.e., zero-shot learning, few-shot learning and persona) on LLMs-based code review automation. In total, we investigate 12 variations of two LLMs-based code review automation (i.e., GPT-3.5 and Magicoder), and compare them with the Guo et al.’s approach and three existing code review automation approaches (i.e., CodeReviewer, TufanoT5 and D-ACT).</p></div><div><h3>Results:</h3><p>The fine-tuning of GPT 3.5 with zero-shot learning helps GPT-3.5 to achieve 73.17%–74.23% higher EM than the Guo et al.’s approach. In addition, when GPT-3.5 is not fine-tuned, GPT-3.5 with few-shot learning achieves 46.38%–659.09% higher EM than GPT-3.5 with zero-shot learning.</p></div><div><h3>Conclusions:</h3><p>Based on our results, we recommend that (1) LLMs for code review automation should be fine-tuned to achieve the highest performance.; and (2) when data is not sufficient for model fine-tuning (e.g., a cold-start problem), few-shot learning without a persona should be used for LLMs for code review automation. Our findings contribute valuable insights into the practical recommendations and trade-offs associated with deploying LLMs for code review automation.</p></div>","PeriodicalId":54983,"journal":{"name":"Information and Software Technology","volume":"175 ","pages":"Article 107523"},"PeriodicalIF":3.8,"publicationDate":"2024-07-11","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.sciencedirect.com/science/article/pii/S0950584924001289/pdfft?md5=526a4187620c208e9aedacd19f66db65&pid=1-s2.0-S0950584924001289-main.pdf","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141728781","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}