Using early LLMs for corpus linguistics: Examining ChatGPT's potential and limitations

Abstract

This study evaluates the extent to which information can be obtained from early Large Language Models (LLMs) for corpus linguistic research. Various tasks were conducted using ChatGPT 3.5, such as generating word frequency lists, collocations, words that fit certain grammatical patterns, and identifying genres. The generations were then compared with the search results from a large-scale general corpus (COCA). While favorable results were not achieved in identifying the genres of words or paragraphs, there was notable congruence in the frequency lists (75.0 %), collocations (42.8 %), and grammatical patterns (53.0 %) for the top 20 items. Even when the generated items did not perfectly match those from COCA, it was evident that high-frequency items were produced. Although LLMs may not be sufficient for rigorous academic research, the results are adequate for discerning overall trends or assisting learners. In addition, the results of this study show that the ability to search at the phrase level is an advantage of using LLMs for corpus research.