A corpus-driven comparative analysis of AI in academic discourse: Investigating ChatGPT-generated academic texts in social sciences

Abstract

Since its release in 2022, ChatGPT has found widespread application across various disciplines. While previous studies on Generative AI’s capabilities have predominantly concentrated on content quality assessments, little attention has been directed toward investigating the model’s linguistic patterns compared to human-generated language. To address this gap, we built two specialized corpora comprised of academic texts in social sciences generated by ChatGPT-4o mini and selected the Elsevier OA CC-BY Corpus as a reference for comparison, with a view to identifying commonalities and differences between AI-generated and human academic language and determining whether academic language instructions improve the model’s output in terms of formal rigor. The findings revealed limitations in ChatGPT’s handling of academic discourse in the following respects: overuse of infrequent “academic” vocabulary, limited use of subordination, and syntactic and semantic homogeneity. Besides, the effect of specific language-oriented prompts is primarily reflected in minor lexical adjustments. This study expands the scope of corpus linguistics research by incorporating AI-generated texts into the analytical framework and lays the groundwork for future improvements in the language model’s genre discrimination.