Is ChatGPT like a nine-year-old child in theory of mind? Evidence from Chinese writing

ChatGPT has been demonstrated to possess significant capabilities in generating intricate human-like text, and recent studies have established that its performance in theory of mind (ToM) tasks is strikingly comparable to a nine-year-old child’s. However, it remains unknown whether ChatGPT outperforms children of this age group in Chinese writing, a task credibly related to ToM. To justify the claim, this study compared ChatGPT with nine-year-old children in making Chinese compositions (i.e., science-themed and nature-themed narratives), aiming to unveil the relative advantages and disadvantages by human writers and ChatGPT in Chinese writing. Based on the evaluative framework comprising of four indices (i.e., fluency, accuracy, complexity, and cohesion) to test writing quality, this study added an often-overlooked index “emotion” to extend the framework. Afterward, we collected 120 writing samples produced by ChatGPT and children and used the confirmatory factor analysis (CFA) and structural equation modelling (SEM) for data analysis and comparison. The results revealed that this age group of children surpassed ChatGPT in fluency and cohesion while ChatGPT transcended the children in accuracy. With respect to complexity, the children exhibited better skills in science-themed writing, but ChatGPT better in nature-themed writing. Most importantly, this study unlocked the pioneering discovery that children display more potent emotional expressions than ChatGPT in Chinese writing, providing an instance of evidence that ChatGPT is really even poorer than a nine-year-old child in ToM to some extent.