Journal of Software-Evolution and Process最新文献

Code generation for users' intent has become increasingly prevalent with the large language models (LLMs). To automatically evaluate the effectiveness of these models, multiple execution-based benchmarks are proposed, including specially crafted tasks, accompanied by some test cases and a ground truth solution. LLMs are regarded as well-performed in code generation tasks if they can pass the test cases corresponding to most tasks in these benchmarks. However, it is unknown whether the test cases have sufficient test adequacy and whether the test adequacy can affect the evaluation. In this paper, we conducted an empirical study to evaluate the test adequacy of the execution-based benchmarks and to explore their effects during evaluation for LLMs. Based on the evaluation of the widely used benchmarks, HumanEval, MBPP, and two enhanced benchmarks HumanEval+ and MBPP+, we obtained the following results: (1) All the evaluated benchmarks have high statement coverage (above 99.16%), low branch coverage (74.39%) and low mutation score (87.69%). Especially for the tasks with higher cyclomatic complexities in the HumanEval and MBPP, the mutation score of test cases is lower. (2) No significant correlation exists between test adequacy (statement coverage, branch coverage and mutation score) of benchmarks and evaluating results on LLMs at the individual task level. (3) There is a significant positive correlation between mutation score-based evaluation and another execution-based evaluation metric ($��A�v�g�P�a�s�s�R�a�t�i�o�$) on LLMs at the individual task level. (4) The existing test case augmentation techniques have limited improvement in the coverage of test cases in the benchmark, while significantly improving the mutation score by approximately 34.60% and also can bring a more rigorous evaluation to LLMs on code generation. (5) The LLM-based test case generation technique (EvalPlus) performs better than the traditional search-based technique (Pynguin) in improving the benchmarks' test quality and evaluation ability of code generation.

In recent years, with the development of the open-source community, various open-source donation platforms have emerged. These platforms effectively alleviate the financial pressures faced by open-source projects through diversified funding sources and flexible donation methods. As one of the most representative open-source donation platforms, Opencollective has garnered widespread attention from both the open-source community and academia. Although Opencollective claims to provide more funding opportunities for open-source projects, the extent to which it effectively addresses the financial challenges faced by these projects remains unclear. While there have been studies on the effectiveness of traditional donation models, research on the effectiveness of emerging donation platforms such as Opencollective is still limited. Given that a large number of open-source projects are urgently seeking donations, understanding the effectiveness of donations through Opencollective is crucial for these projects. To address this gap, we have made an early step in this direction. This paper conducts a comprehensive study on the effectiveness of donations through the Opencollective, employing a combination of quantitative and qualitative analysis and identifies the following key findings: (1) Opencollective attracts a diverse group of participants, including individual donors, sponsors, contributors, and project managers, with individual donors constituting the largest group. Most donations are concentrated in the range of $5 to $10, indicating that the platform largely relies on small but frequent donations from individuals. (2) Only about 26.61% of open-source projects receive donations through Opencollective, with approximately 64.38% of these projects receiving a total donation amount of less than $50,000. The likelihood of receiving donations increases with project scale, maturity and the number of stars. Among projects that have received donations, larger projects with stronger social media promotion, greater attention and more issues are more likely to receive additional donations. (3) The positive impact of donations on project development and spend activities is significant only in the short term, with no notable long-term effects. In contrast, donations do not have a significant short-term impact on community engagement. Although the long-term effect is slightly positive, it is not statistically significant. (4) The main shortcomings of Opencollective include insufficient project management and collaboration features, inadequate user experience and interface design, high transaction fees, and a lack of transparency in fund allocation and usage. Our findings provide significant theoretical support and practical recommendations for the effectiveness of emerging donation platforms and the sustainable development of open-source projects.

In this study, a maturity-level determination and assessment method developed for companies operating in the software industry to perform TMMi Levels 2 and 3 assessments in-house, with the goal of improving testing processes, is presented. With this method, it is aimed to help companies to conduct their own self-assessments and improve their testing processes before participating in high-budget audits. The validity of this method was tested in practice for TMMi Levels 2 and 3 assessments. Companies can prepare for a formal TMMi audit by using the test maturity-level determination methodology developed in this study, or they can simply improve their processes to produce higher quality products. Additionally, if they already have TMMi certification, they can regularly self-audit to ensure continuity and compliance. Demonstrating how TMMi can be applied in practice and providing a guide to test process maturity determination are key contributions of this work.

With the increasingly frequent attacks on software systems, software security is an issue that must be addressed. Within software security, automated detection of software vulnerabilities is an important subject. Most existing vulnerability detectors rely on the features of a single code type (e.g., source code or intermediate representation [IR]), which may lead to both the global features of the code slices and the memory operation information not being captured or considered. In particular, vulnerability detection based on source-code features cannot usually include some macro or type definition content. In this paper, we propose a vulnerability-detection method that combines the semantic features of source code and the low level virtual machine (LLVM) IR. Our proposed approach starts by slicing (C/C++) source files using improved slicing techniques to cover more comprehensive code information. It then extracts semantic information from the LLVM IR based on the executable source code. This can enrich the features fed to the artificial neural network (ANN) model for learning. We conducted an experimental evaluation using a publicly-available dataset of 11,381 C/C++ programs. The experimental results show the vulnerability-detection accuracy of our proposed method to reach over 96% for code slices generated according to four different slicing criteria. This outperforms most other compared detection methods.

Contemporary program protection methods focus on safeguarding either program generation, storage, or execution; however, no unified protection strategy exists for ensuring the security of a full program lifecycle. In this study, we combine the static security of program generation with the dynamic security of process execution and propose a novel program logic consistency security property. An encryption core processing (ECP) architecture is presented that provides coprocessor solutions to protect the program logic consistency at the granularity of instructions and data flows. The new authenticated encryption mode in the architecture uses the offset value of the program's instructions and data in relation to the segment-based address as its encryption parameters. Lightweight cryptographic primitives are adopted to ensure that the hardware burden added by the ECP is limited, especially under $��\times �$64 architectures. We prove that the proposed scheme in the ECP architecture satisfies indistinguishability under chosen plaintext attack and demonstrate the effectiveness of the architecture against various attacks. Additionally, a theoretical performance analysis is provided for estimating the overhead introduced by the ECP architecture.

A software ecosystem is a complex system that allows developers to cooperate with each other. Community is a universal and important topological property of networks. Detecting the communities of the software ecosystem is of great significance for analyzing its structural characteristics, discovering its hidden patterns, and predicting its behavior. Traditional community detection algorithms of complex networks are mostly for undirected networks. For the social network, the direction of information dissemination between developers cannot be ignored. In addition, the existing algorithms of community detection usually only consider direct influence between individuals while neglecting indirect relationships. To solve these problems, this paper presents a community detection method based on a two-step information dissemination model for the software ecosystem. First, a two-step information dissemination model is established to calculate the information gain of nodes. Second, a ranking method of developers' comprehensive influence is given through their influence vectors and information gains. Finally, communities are detected by taking the influential nodes as the cluster centers and the probability of information dissemination as the clustering direction. The proposed method is applied to community detection of typical software ecosystems in GitHub. The experimental results show that our method has good performance in the identification of community structure.