Security Analysis of Large Language Models on API Misuse Programming Repair

Abstract

Application programming interface (API) misuse refers to misconceptions or carelessness in the anticipated usage of APIs, threatening the software system’s security. Moreover, API misuses demonstrate significant concealment and are challenging to uncover. Recent advancements have explored enhanced LLMs in a variety of software engineering (SE) activities, such as code repair. Nonetheless, the security implications of using LLMs for these purposes remain underexplored, particularly concerning the issue of API misuse. In this paper, we present an empirical study to observe the bug-fixing capabilities of LLMs in addressing API misuse related to monitoring resource management (MRM API misuse). Initially, we propose APImisRepair, a real-world benchmark for repairing MRM API misuse, including buggy programs, corresponding fixed programs, and descriptions of API misuse. Subsequently, we assess the performance of several LLMs using the APImisRepair benchmark. Findings reveal the vulnerabilities of LLMs in repairing MRM API misuse and find several reasons, encompassing factors such as fault localization and a lack of awareness regarding API misuse. Additionally, we have insights on improving LLMs in terms of their ability to fix MRM API misuse and introduce a crafted approach, APImisAP. Experimental results demonstrate that APImisAP exhibits a certain degree of improvement in the security of LLMs.