Unveiling the Role of GPT-4 in Solving LeetCode Programming Problems

Abstract

The landscape of programming education is undergoing a transformative shift in the era of AI and machine learning. This research delves into the role of GPT-4, a state-of-the-art language model, in solving intermediate-level programming problems, focusing on the renowned LeetCode platform. For this, the work employs different programming problems from two LeetCode contests, providing a comprehensive evaluation of GPT-4's capabilities. The results reveal intriguing patterns in the model's behavior. Initial attempts, when provided with all inputs simultaneously, exhibit high accuracy, but subsequent attempts show consistent fluctuations, rarely surpassing the accuracy of the first attempt. Upon closer examination, a distinct pattern emerges in GPT-4's problem-solving approach, where the model iteratively refines its solutions, incorporating corrections in subsequent attempts. However, the lack of a historical context for past attempts raises questions about the model's attention span and its ability to rectify mistakes. Notably, GPT-4 consistently fails on the same test case with the same generated output, suggesting a potential limitation in addressing specific challenges. But, on leveraging human assistance to AI tools, the observations and patterns from the incorrect codes can be drawn and required adjustments to rectify the codes can be made. A direct result of this is observed in the increased success rate in problem-solving by students, rising from 68% in the moderate learning stage to 92% in the advanced learning stage. Hence, the presented work proposes a human-supervised methodology to leverage the AI-assisted code generation and employs that in improving the effectiveness of AI-assisted teaching–learning process.