Journal of Software-Evolution and Process最新文献

Large language models (LLMs) have recently achieved significant success across various application domains, garnering substantial attention from different communities. Unfortunately, many faults still exist that LLMs cannot properly predict. Such faults will harm the usability of LLMs in general and could introduce safety issues in reliability-critical systems such as autonomous driving systems. How to quickly reveal these faults in real-world datasets that LLMs could face is important but challenging. The major reason is that the ground truth is necessary but the data labeling process is heavy considering the time and human effort. To handle this problem, in the conventional deep learning testing field, test selection methods have been proposed for efficiently evaluating deep learning models by prioritizing faults. However, despite their importance, the usefulness of these methods on LLMs is unclear and underexplored. In this paper, we conduct the first empirical study to investigate the effectiveness of existing test selection methods for LLMs. We focus on classification tasks because most existing test selection methods target this setting and reliably estimating confidence scores for variable-length outputs in generative tasks remains challenging. Experimental results on four different tasks (including both code tasks and natural language processing tasks) and four LLMs (e.g., LLaMA3 and GPT-4) demonstrated that simple methods such as Margin perform well on LLMs, but there is still a big room for improvement. Based on the study, we further propose MuCS, a prompt Mutation-based prediction Confidence Smoothing framework to boost the test selection capability for LLMs specifically on classification tasks. Concretely, multiple prompt mutation techniques have been proposed to help collect diverse outputs for confidence smoothing. The results show that our proposed framework significantly enhances existing methods with test relative coverage improvement by up to 70.53%.

Strong cybersecurity procedures are now more important than ever due to the increased reliance on remote workers. Given the dynamic nature of cyber threats and the necessity of preventative actions, this paper highlights the vital significance of thorough cybersecurity awareness training for remote workers. A customized cybersecurity awareness training model can improve distributed team preparedness and decrease cyberattacks. Organizations should institute regular security awareness programs to educate distributed teams on emerging cyber threats. Vendor businesses should prioritize security education to prevent cyberattacks and protect sensitive data. Our proposed model aims to improve distributed team members' preparedness against cyber threats, enabling organizations to safeguard remote work settings effectively. Our systematic literature review identified key cybersecurity factors, synthesized into 12 groups, including “Unified Governance Framework” and “Secure Mind Initiative.”

Merge conflicts arise when multiple developers simultaneously modify the same part of a codebase and attempt to merge their changes. These conflicts occur because the version control system (VCS) cannot automatically determine which changes should take precedence. Resolving such conflicts involves manually reviewing the conflicting changes and deciding how to integrate them to maintain a functional and coherent codebase. This process is often time-consuming, complex, and prone to errors. Consequently, the software engineering community has focused on predicting merge conflicts to warn developers early and allow them to address conflicts before they escalate. Despite several efforts to predict merge conflicts, no perfect solution has been identified. Fortunately, many machine learning techniques have demonstrated potential in improving prediction performance across various contexts. This study aims to empirically investigate the effectiveness of stacking heterogeneous ensembles in enhancing merge conflict prediction performance. We empirically compared the prediction performance of the following individual models: decision trees (DT); support vector machine (SVM) with a linear kernel; naive Bayes (NB) with Bernoulli, Gaussian, and Multinomial variants; logistic regression (LR); multilayer perceptron (MLP); stochastic gradient descent (SGD); and k-nearest neighbors (KNN). Additionally, we evaluated three heterogeneous stacking ensembles: Stack-DT, Stack-SVM, and Stack-LR, which were constructed using the aforementioned individual models as base models. We utilized gain ratio (GR) to identify the most important technical and social features for predicting merge conflicts and assessed the impact of using only these important features on the performance of both individual and stacking models. The study revealed variability in the performance of individual models, with DT demonstrating the best predictive performance among them. Heterogeneous stacking ensembles demonstrated potential to enhance merge conflict prediction, with Stack-SVM emerging as the top-performing model. GR analysis highlighted the importance of both social and technical features in predicting merge conflicts. However, using only the most important features identified by GR led to a decline in the performance of most models compared to using all features. Heterogeneous stacking ensembles significantly improve prediction performance over individual models. Both social and technical features are important in predicting merge conflicts, and utilizing the full set of features instead of only the most important ones generally yields better results.

Security threats in web applications have increasingly become a major concern, particularly as modern web systems grow more complex and interconnected. Addressing these security challenges requires a comprehensive understanding of how threats are distributed across different phases of the software development life cycle (SDLC) and how various threat categories map to specific SDLC stages. Despite significant research into software security, a systematic and structured review focusing on the hierarchical relationships between SDLC phases, security threat categories, and specific threats remains scarce. This paper aims to fill this gap by conducting a clustering-based systematic review of security threats in web applications. Using data from existing literature on software security threats, we applied hierarchical clustering, K-means analysis, and co-occurrence mapping to identify relationships between SDLC phases (Level 1), security threat categories (Level 2), and specific security threats (Level 3). The findings show that the development phase presents the highest risk, more so to threats like weaknesses in architectural security design and input validation issues. Using clustering techniques, we showed how some of the threats appeared in more than one SDLC stage and classified them within the categories of threats most closely associated with the SDLC stage. Taking into account these factors, we propose recommendations for software development process stakeholders allowing for the implementation of more consistent strategies of threat mitigation through the entire SDLC. Considering these observations, it can be concluded that there is an acute deficiency in development for globalization of software security measures towards web applications to control future security threats.

An operating system is the essence of software, serving as the foundation for the operation of various application software. The security of the operating system is crucial for national informatization construction. Data indicate that many cybersecurity incidents result from exploiting security vulnerabilities in the operating system. Linux is currently the most widely used open-source operating system, with thousands of Common Vulnerabilities and Exposures (CVEs) related to Linux systems reported each year. Therefore, research and prevention of vulnerabilities in the Linux system are particularly important. To gain a better understanding of the characteristics of Linux system vulnerabilities, this paper leverages knowledge in the field of software security to analyze nearly 10,000 historical vulnerability data in two core systems of Linux: Linux Kernel and Debian Linux. The study explores the evolutionary patterns of vulnerability characteristics. Specific research contents include the following: (1) data collection and cleaning of vulnerability data in Linux Kernel and Debian Linux systems; (2) cross-statistical analysis of structured data features in vulnerability reports; (3) unstructured data characteristics mining in vulnerability reports based on domain knowledge; (4) analysis of the evolution of vulnerability characteristics. This paper provides empirical lessons and guidance for Linux system vulnerabilities to assist practitioners and researchers in better preventing and detecting vulnerabilities in Linux and Linux-based systems.

Textual vulnerability descriptions (TVDs) in repositories like NVD and IBM X-Force Exchange are essential for security engineers managing vulnerabilities. Engineers typically search for key aspects in TVDs using specific phrases, but with multiple expressions for each aspect, retrieving all relevant records is challenging. We propose a label-based retrieval framework that classifies key aspects and retrieves TVDs by their broader categories. Given the large data volume, manual labeling is infeasible, making unsupervised classification critical. However, short labels and repeated words diminish semantic clarity, affecting classification accuracy. We introduce Unsupervised Classification through Label Profile (UCLP), which expands label semantics through label profiles inspired by recommendation systems. We construct profiles using neural network weights and apply TF-IDF to calculate similarities, smoothing distributions with an arctangent function. Results show that UCLP significantly outperforms four benchmarks, raising accuracy from 68.3% to 78.9% and improving three real-world applications.

Heterogeneous defect prediction (HDP) plays a crucial role in software engineering by enabling the early detection of software defects across projects with heterogeneous feature spaces. Recently, some mixed-project HDP (MP-HDP) methods have been proposed, which have demonstrated modest improvements in HDP performance. Nevertheless, existing MP-HDP approaches fail to address feature redundancy and distribution inconsistency simultaneously. To overcome these limitations, this paper proposes a novel MP-HDP approach, UFR-OSFA, based on unified feature representation and oppositional structural feature alignment. Concretely, UFR-OSFA first unifies these features by reducing the distribution differences between source and target projects through matching common features and the Hungarian algorithm based on the Kolmogorov–Smirnov (KS) test. Subsequently, utilizing a generator and two classifiers with oppositional structures, UFR-OSFA separates the features of the source project and clusters those of the target project, addressing the issue of conditional distribution mismatch and enhancing the model's generalization ability in the target project. Extensive experiments on 23 projects from five datasets demonstrate that the proposed approach performs better or comparably to baseline methods.