Journal of Systems and Software最新文献

{"title":"Exploring challenges in test mocking: Developer questions and insights from StackOverflow","authors":"Mumtahina Ahmed ,&nbsp;Md Nahidul Islam Opu ,&nbsp;Chanchal Roy ,&nbsp;Sujana Islam Suhi ,&nbsp;Shaiful Chowdhury","doi":"10.1016/j.jss.2025.112748","DOIUrl":"10.1016/j.jss.2025.112748","url":null,"abstract":"<div><div>Mocking is a common unit testing technique that is used to simplify tests, reduce flakiness, and improve coverage by replacing real dependencies with simplified implementations. Despite its widespread use in Open Source Software (OSS) projects, there is limited understanding of how and why developers use mocks and the challenges they face. In this study, we have analyzed 25,302 questions related to <em>Mocking</em> on StackOverflow to identify the challenges faced by developers. We have used Latent Dirichlet Allocation (LDA) for topic modeling, identified 30 key topics, and grouped the topics into five key categories. Consequently, we analyzed the annual and relative probabilities of each category to understand the evolution of mocking-related discussions. Trend analysis reveals that categories such as <em>Mocking Techniques</em> and <em>External Services</em> have remained consistently dominant, highlighting evolving developer priorities and ongoing technical challenges. While the questions on <em>Theoretical</em> category declined after 2010, posts regarding <em>Error Handling</em> grew notably from 2009.</div><div>Our findings also show an inverse relationship between a topic’s popularity and its difficulty. Popular topics like <em>Framework Selection</em> tend to have lower difficulty and faster resolution times, while complex topics like <em>HTTP Requests and Responses</em> are more likely to remain unanswered and take longer to resolve. Additionally, we evaluated questions based on the answer status- successful, ordinary, or unsuccessful, and found that topics such as <em>Framework Selection</em> have higher success rates, whereas tool setup and Android-related issues are more often unresolved. A classification of questions into <em>How, Why, What</em>, and <em>Other</em> revealed that over 64 % are <em>How</em> questions, particularly in practical domains like file access, APIs, and databases, indicating a strong need for implementation guidance. <em>Why</em> questions are more prevalent in error-handling contexts, reflecting conceptual challenges in debugging, while <em>What</em> questions are rare and mostly tied to theoretical discussions. These insights offer valuable guidance for improving developer support, tooling, and educational content in the context of mocking and unit testing.</div></div>","PeriodicalId":51099,"journal":{"name":"Journal of Systems and Software","volume":"235 ","pages":"Article 112748"},"PeriodicalIF":4.1,"publicationDate":"2025-12-31","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145928355","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"LeCov: Multi-level testing criteria for large language models","authors":"Xuan Xie ,&nbsp;Jiayang Song ,&nbsp;Yuheng Huang ,&nbsp;Da Song ,&nbsp;Felix Juefei-Xu ,&nbsp;Lei Ma","doi":"10.1016/j.jss.2025.112763","DOIUrl":"10.1016/j.jss.2025.112763","url":null,"abstract":"<div><div>Large Language Models (LLMs) are widely used in many different domains, but because of their limited interpretability, there are questions about how trustworthy they are in various perspectives, e.g., truthfulness and toxicity. Recent research has started developing testing methods for LLMs, aiming to uncover untrustworthy issues, i.e., defects, before deployment. However, systematic and formalized testing criteria are lacking, which hinders a comprehensive assessment of the extent and adequacy of testing exploration. To mitigate this threat, we propose a set of multi-level testing criteria, LeCov, for LLMs. The criteria consider three crucial LLM internal components, i.e., the attention mechanism, feed-forward neurons, and uncertainty, and contain nine types of testing criteria in total. We apply the criteria in two scenarios: test prioritization and coverage-guided testing. The experiment evaluation, on five models and four datasets, demonstrates the usefulness and effectiveness of LeCov.</div></div>","PeriodicalId":51099,"journal":{"name":"Journal of Systems and Software","volume":"235 ","pages":"Article 112763"},"PeriodicalIF":4.1,"publicationDate":"2025-12-30","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145928356","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"A reference architecture for ethical-aware autonomous systems","authors":"Marco Autili ,&nbsp;Martina De Sanctis ,&nbsp;Paola Inverardi ,&nbsp;Mashal Afzal Memon ,&nbsp;Patrizio Pelliccione ,&nbsp;Sara Pettinari","doi":"10.1016/j.jss.2025.112749","DOIUrl":"10.1016/j.jss.2025.112749","url":null,"abstract":"<div><div><em>Background.</em> Autonomous systems, whether AI-enabled or not, are ubiquitous and pervasive in our daily lives. While their adoption and use bring many benefits, they also pose significant ethical challenges. <em>Objective.</em> The objective of this work is to contribute a reference architecture for ethical-aware autonomous systems, focusing on their interaction and collaboration with humans, being them proactive, reactive or passive in the interaction with the systems. <em>Method.</em> To define the architecture, we analyzed scientific papers in the field, guidelines and recommendations, as well as laws and regulations. We then applied this acquired knowledge to build the reference architecture. The results of this work were validated through expert interviews and a scenario-based evaluation. <em>Results.</em> We contribute (i) a definition of ethical-aware autonomous systems, (ii) requirements for ethical-aware autonomous systems, and (iii) a reference architecture for ethical-aware autonomous systems. Our reference architecture is intended to help system and software engineers to design autonomous or intelligent systems that should interact and operate with humans in ethically sensitive contexts such as healthcare, social robotics, and assistive technologies. <em>Conclusion.</em> We believe that this work will assist software architects and engineers in designing and developing autonomous systems that should interact and collaborate with humans while respecting values important to individuals, society, and the environment.</div></div>","PeriodicalId":51099,"journal":{"name":"Journal of Systems and Software","volume":"235 ","pages":"Article 112749"},"PeriodicalIF":4.1,"publicationDate":"2025-12-30","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145886520","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Inter-organizational collaborations in open-source software ecosystems","authors":"Roland Robert Schreiber ,&nbsp;Thomas Wieland","doi":"10.1016/j.jss.2025.112765","DOIUrl":"10.1016/j.jss.2025.112765","url":null,"abstract":"<div><h3>Context</h3><div>Open-source software (OSS) ecosystems are pivotal to modern software development by fostering innovation through inter-organizational collaboration. While becoming increasingly important, research on the patterns and dynamics of this collaboration in OSS remains limited.</div></div><div><h3>Objective</h3><div>This study investigates inter-organizational collaboration within different OSS ecosystems, focusing on identifying key contributing and influential organizations, analyzing collaboration patterns, and understanding their evolution over time.</div></div><div><h3>Method</h3><div>An exploratory case study with a mixed-methods approach was conducted, with data collected from five prominent OSS ecosystems on GitHub (React, Vue, TensorFlow, Bootstrap, and Flutter) involving 9947 developers and 339 organizations over ten years. Both qualitative and quantitative analyses were performed, including software repository mining and applying different levels (micro/macro) and dimensions (rational/structural/functional) of social network analysis (SNA) to examine collaboration patterns.</div></div><div><h3>Results</h3><div>Key organizations, such as Alphabet, TensorFlow, Meta, Nvidia, Intel, MobileIron, IBM, and AMD, emerged as significant contributors, acting as central hubs for collaboration within OSS ecosystems. The collaboration networks showed an initial phase of rapid growth followed by stabilization, indicating project maturation. Notably, competing firms were found to contribute to the same ecosystems, underpinning the inherently cooperative, reciprocal, and multiplex nature of OSS development.</div></div><div><h3>Conclusion</h3><div>The findings highlight the pivotal role of top-contributing organizations in fostering collaboration and driving innovation within OSS ecosystems. Understanding these dynamics provides valuable insights for developing strategies to enhance the effectiveness, scalability, and long-term sustainability of OSS ecosystems, benefiting both individual contributors and participating organizations.</div></div>","PeriodicalId":51099,"journal":{"name":"Journal of Systems and Software","volume":"235 ","pages":"Article 112765"},"PeriodicalIF":4.1,"publicationDate":"2025-12-30","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145886521","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Enabling direct manipulation of plain-text output for template programs","authors":"Tao Zan ,&nbsp;Xiao He ,&nbsp;Zhenjiang Hu","doi":"10.1016/j.jss.2025.112754","DOIUrl":"10.1016/j.jss.2025.112754","url":null,"abstract":"<div><div>In modern software development, templates are widely used to generate artifacts, enhancing productivity and consistency. However, developers often face a tedious run-modify-run loop to verify template correctness, as outputs are only inspectable after execution. This issue is exacerbated when templates involve complex logic, highlighting the need for a solution that supports both artifact generation and user-friendly modifications. We propose <span>BIT2</span>, a framework enabling direct manipulation of plain-text output to modify template programs. A key challenge is that plain-text output lack structural information, making it difficult to map updates back to the template. <span>BIT2</span> addresses this by generating computation-structured output, preserving the underlying logic, and supporting update operations (insertion, replacement, deletion) with carefully designed fusion semantics. Our implementation, written in TypeScript (18,000 lines of code), is available as an NPM module, complemented by a VSCode plugin, LiveT, for seamless template editing. Experimental results demonstrate that <span>BIT2</span> achieves efficient evaluation and supports diverse update scenarios, significantly enhancing expressiveness in template development.</div></div>","PeriodicalId":51099,"journal":{"name":"Journal of Systems and Software","volume":"235 ","pages":"Article 112754"},"PeriodicalIF":4.1,"publicationDate":"2025-12-30","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145928352","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"AI in GUI-based testing: A survey of techniques, tools, and perceived advantages and limitations","authors":"Domenico Amalfitano ,&nbsp;Riccardo Coppola ,&nbsp;Damiano Distante ,&nbsp;Filippo Ricca","doi":"10.1016/j.jss.2025.112751","DOIUrl":"10.1016/j.jss.2025.112751","url":null,"abstract":"<div><div><em>Background:</em> The adoption of Artificial Intelligence (AI) techniques in Software Testing (ST) has grown rapidly, particularly in response to the increasing complexity of modern systems. In GUI-based testing, AI is often cited as a promising means to automate repetitive tasks and improve testing efficiency. However, the actual use of AI in this domain remains underexplored through systematic empirical investigation.</div><div><em>Objective:</em> This study aims to analyze how AI is adopted in GUI-based testing, identifying the techniques and tools employed, the testing activities they support, and the perceived benefits and limitations.</div><div><em>Method:</em> We conducted a large-scale survey involving 107 participants from both academia and industry. The survey focuses on three core testing activities: test case definition, test oracle design, and test case optimization. It extends a prior study based on interviews with 45 industry practitioners.</div><div><em>Results:</em>Findings show that AI is primarily used to support test case definition, with techniques such as Natural Language Processing, Optimization, and Large Language Models (LLMs) being the most common. AI also provides support in test oracle design, where image processing and knowledge representation play key roles, and in test suite optimization, through the use of supervised learning, reinforcement learning, and search-based techniques.</div><div><em>Conclusion:</em> The paper identifies ongoing challenges and outlines future directions, including the need for transparent AI tools, guidelines for LLM integration, and the deployment of a continuously open survey to monitor trends in AI adoption over time.</div></div>","PeriodicalId":51099,"journal":{"name":"Journal of Systems and Software","volume":"235 ","pages":"Article 112751"},"PeriodicalIF":4.1,"publicationDate":"2025-12-29","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145928469","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Software refactoring research with large language models: A systematic literature review","authors":"Sofia Martinez ,&nbsp;Luo Xu ,&nbsp;Mariam Elnaggar,&nbsp;Eman Abdullah Alomar","doi":"10.1016/j.jss.2025.112762","DOIUrl":"10.1016/j.jss.2025.112762","url":null,"abstract":"<div><div>Background: Code refactoring is the improvement of code internally without changing the external functionalities of the program. Due to its exhaustive nature, developers often avoid manually refactoring code. Researchers have since looked into utilizing Large Language Models (LLMs) to automate the task of refactoring.</div><div>Aim and Method: Despite the promising results, there is a lack of clear understanding of LLMs’ effectiveness in automated refactoring. In order to address this issue, we conducted a Systematic Literature Review (SLR) of 50 primary studies. We categorized the studies into different refactoring methods studied, prompt engineering and techniques conducted, LLM tools used, languages used, and datasets used. We touched upon the benchmarks each studies had used, how accurate LLM-generated refactorings are, and the challenges that this field faces currently.</div><div>Result: From our literature review we found that: (i) There are various tools that different studies use to enhance and study LLM-driving refactoring, with tools that were used to detect code smells, generate code bases, and compare refactoring outcomes. (ii) Various datasets were collected from multiple open-source projects in multiple programming languages for analysis. These platforms included GitHub, Apache, and F-Droid, with the most popular language collected and analyzed being Java. (iii) One-Shot, Few-Shot, Context-Specific, and Chain-of-Thought prompting methods have been shown to be the most effective depending on the language used. In some instances, being capable of reducing code smell by up to 89%. (iv) The definition of “Accuracy” varies significantly across the literature surveyed, as this depends on the context of the study, thus calling for a need to have a standardized measurement for Accuracy. (v) The most often mentioned code smells were Large Class and Long Method, with a lot of studies also not specifying, while the most often applied refactoring type is Extract Method, showing promising results in using LLM to perform this refactoring type. (vi) When working with LLMs, they often generate erroneous code, struggle with more complex refactoring, and often misunderstand the developers’ requests and miss refactoring requests.</div><div>Conclusion: Our study serves to be a collection of knowledge on the topic of LLM refactoring found by various other studies, and to highlight any issues that are often missed by the researchers. We hope to empower and guide the future development of LLM-driven refactoring with our findings.</div></div>","PeriodicalId":51099,"journal":{"name":"Journal of Systems and Software","volume":"235 ","pages":"Article 112762"},"PeriodicalIF":4.1,"publicationDate":"2025-12-27","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145886522","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Mitigation strategies for confidentiality violations in software architecture using ranked feature importance","authors":"Nils Niehues ,&nbsp;Sebastian Hahner ,&nbsp;Robert Heinrich","doi":"10.1016/j.jss.2025.112761","DOIUrl":"10.1016/j.jss.2025.112761","url":null,"abstract":"<div><div>A quality attribute like confidentiality is critical to trustworthy software but unfortunately, very challenging to ensure. This is because modern software systems are complex and interconnected. Architecture-based confidentiality analysis enables the early detection of violations, helping to mitigate risks before deployment. However, uncertainty in software systems and their environments complicates precise and comprehensive architectural analysis. Additionally, the complexity of software models and the exponential growth of uncertainty scenarios pose significant challenges for automated mitigation, often leaving software architects to resolve confidentiality violations manually, a process that is both time-intensive and error-prone.</div><div>In this paper, we extend our machine-learning-based approach to mitigate confidentiality violations. Specifically, we introduce a novel mitigation strategy inspired by TCP Congestion Control, as well as a strategy that capitalizes on clustering techniques to dynamically adjust batch sizes. Our evaluation on three real-world software architectures demonstrates that our extended approach can mitigate confidentiality violations while outperforming the state-of-the-art. Whereas previously the upper limit was 60 times runtime reduction, now we achieve 2298 times reduction, with the median being an elevenfold reduction. Our statistical analysis confirms that the added TCP-inspired strategy is significantly cheaper than the state-of-the-art baseline (Friedman test <span><math><mrow><mi>p</mi><mo>=</mo><mo>.</mo><mn>025</mn></mrow></math></span> and Nemenyi post hoc test <span><math><mrow><mi>p</mi><mo>=</mo><mo>.</mo><mn>039</mn></mrow></math></span>), while also having a strong practical impact (Kendall’s W <span><math><mrow><mo>=</mo><mn>0.721</mn></mrow></math></span>). This extended work deepens our understanding of the nature of uncertainty and also of the techniques optimally suited to mitigating the violations caused by uncertainties. It takes us one step closer to designing trustworthier systems.</div></div>","PeriodicalId":51099,"journal":{"name":"Journal of Systems and Software","volume":"235 ","pages":"Article 112761"},"PeriodicalIF":4.1,"publicationDate":"2025-12-26","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145928470","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Ethics label for digital systems to promote transparency and user awareness","authors":"Marco Autili ,&nbsp;Riccardo Corsi ,&nbsp;Martina De Sanctis ,&nbsp;Paola Inverardi ,&nbsp;Patrizio Pelliccione","doi":"10.1016/j.jss.2025.112752","DOIUrl":"10.1016/j.jss.2025.112752","url":null,"abstract":"<div><div>Modern digital systems pose risks to humans, society, and the environment. There is a flourishing of guidelines, recommendations, laws, and regulations, but also of standards that try to regulate and alleviate the lack of good practices for the governance, management, and quality of AI systems. However, having just a mark certifying that the system passed some checks is not enough and is fragile to the ethics washing problem. The objective of this work is to go beyond compliance to standards toward an ethics label that enables users to understand the impact of systems on human, societal, and environmental values, both during system development and usage. To build the ethics label, we analyze guidelines, recommendations, laws and regulations, and quality standards. We validate the proposed ethics label through (i) a proof-of-concept application to the social assistive robotics domain, and (ii) interviews with experts from both academia and industry. We contribute an ethics label for modern digital systems that promote transparency and user awareness, and enable users to select systems that meet their subjective ethical preferences. We discuss the problem of ethics washing and propose an ethics label as a means to foster transparency and raise user awareness.</div></div>","PeriodicalId":51099,"journal":{"name":"Journal of Systems and Software","volume":"235 ","pages":"Article 112752"},"PeriodicalIF":4.1,"publicationDate":"2025-12-26","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"146038341","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"A systematic literature review of software engineering research on Jupyter notebook","authors":"Md Saeed Siddik ,&nbsp;Hao Li ,&nbsp;Cor-Paul Bezemer","doi":"10.1016/j.jss.2025.112758","DOIUrl":"10.1016/j.jss.2025.112758","url":null,"abstract":"<div><div><em>Context</em>: Jupyter Notebook has emerged as a versatile tool that transforms how researchers, developers, and data scientists conduct and communicate their work. As the adoption of Jupyter notebooks continues to rise, so does the interest from the software engineering research community in improving the software engineering practices for Jupyter notebooks.</div><div><em>Objective</em>: The purpose of this study is to analyze trends, gaps, and methodologies used in software engineering research on Jupyter notebooks.</div><div><em>Method</em>: We selected 199 relevant publications up to September 2025, following established systematic literature review guidelines. We explored publication trends, categorized them based on software engineering topics, and reported findings based on those topics.</div><div><em>Results</em>: The most popular venues for publishing software engineering research on Jupyter notebooks are related to human-computer interaction instead of traditional software engineering venues. Researchers have addressed a wide range of software engineering topics on notebooks, such as code reuse, readability, and execution environment. Although reusability is one of the research topics for Jupyter notebooks, only 82 of the 199 studies can be reused based on their provided URLs. Additionally, most replication packages are not hosted on permanent repositories for long-term availability and adherence to open science principles.</div><div><em>Conclusion</em>: Solutions specific to notebooks for software engineering issues, including testing, refactoring, and documentation, are underexplored. Future research opportunities exist in automatic testing frameworks, refactoring clones between notebooks, and generating group documentation for coherent code cells.</div></div>","PeriodicalId":51099,"journal":{"name":"Journal of Systems and Software","volume":"235 ","pages":"Article 112758"},"PeriodicalIF":4.1,"publicationDate":"2025-12-25","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145852497","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}