Evaluate Chat‐GPT's programming capability in Swift through real university exam questions

Abstract

In this study, we evaluate the programming capabilities of OpenAI's GPT‐3.5 and GPT‐4 models using Swift‐based exam questions from a third‐year university course. The results indicate that both GPT models generally outperform the average student score, yet they do not consistently exceed the performance of the top students. This comparison highlights areas where the GPT models excel and where they fall short, providing a nuanced view of their current programming proficiency. The study also reveals surprising instances where GPT‐3.5 outperforms GPT‐4, suggesting complex variations in AI model capabilities. By providing a clear benchmark of GPT's programming skills in an academic context, our research contributes valuable insights for future advancements in AI programming education and underscores the need for continued development to fully realize AI's potential in educational settings.