Exploring the Use of a Large Language Model for Inductive Content Analysis in a Discourse Network Analysis Study

Abstract

Large language models show promising capability in some qualitative content analysis tasks; however, research reporting their performance in identifying initial codes that underpin subsequent analysis is scarce. This paper explores the suitability of GPT-4 to assist in building a codebook for a discourse network analysis (DNA) of a recent alcohol policy reform. DNA is a codebook-driven approach to identifying groupings of actors who use similar policy framings. The paper uses GPT-4 to identify initial codes (‘concepts’) and related quotes in 108 news articles and interviews. The results produced by GPT-4 are compared to a codebook prepared by researchers. GPT-4 identified over two-thirds of the concepts found by the researchers, and it was highly accurate in screening out a large volume of irrelevant media items. However, GPT-4 also provided many irrelevant concepts that required researcher review and removal. The discussion reflects on the implications for using GPT-4 in codebook preparation for DNA and other situations, including the need for human involvement and sample testing to understand its strengths and limitations, which may limit efficiency gains.