Exploring the affordances of generative AI large language models for stance and engagement in academic writing

Abstract

Large pre-trained models like ChatGPT demonstrate remarkable capabilities in generating coherent text across various domains, posing serious implications for teaching academic writing, given the potential for student plagiarism and reliance on software for developing writing skills. However, the linguistic properties and strategies these models employ remain largely unexplored. We investigate how three available large language models (LLMs) express stance and engage with readers in their writing, providing insights into their abilities to produce contextually appropriate and discipline-specific academic writing. 30 academic essays produced by each model were compared with those of human writers on identical topics using detailed prompts, before annotating each text for stance and engagement following Hyland's (2005) taxonomy. Results indicate that LLMs generally use a narrower and more repetitive range of stance and engagement features than human writers, with significant variation also across each LLM. Disciplinary use of stance and engagement is largely in line with human writing except for the philosophy discipline. Implications for teaching academic writing are discussed, particularly regarding identifying potential LLM-related plagiarism and inconsistencies in academic stance and engagement.