Overview and confirmatory and exploratory factor analysis of AI literacy scale

Comprehensive concepts of AI literacy (AIL) and valid measures are essential for research (e.g., intervention studies) and practice (e.g., personnel selection/development) alike. To date, several scales have been published, sharing standard features but differing in some aspects. We first aim to briefly overview instruments identified from unsystematic literature research in February 2023. We identified four scales and one collection of items. We describe the instruments and compare them. We identified common themes and overlaps in the instruments and developmental procedure. We also found differences regarding scale development procedures and latent dimensions. Following this literature research, we came to the conclusion that the literature on AI literacy measurement was fragmented, and little effort was undertaken to integrate different AI literacy conceptualizations. The second focus of this study is to test the factorial structures of existing AIL measurement instruments and identify latent dimensions of AIL across all instruments. We used robust maximum-likelihood confirmatory factor analysis to test factorial structures in a joint survey of all AIL items in an English-speaking online sample (N=219). We found general support for all instruments' factorial structures with minor deviations from the original factorial structures for some of the instruments. In a second analysis step, to address the issue of fragmented research on AI literacy conceptualization and measurement, we used principal axis exploratory factor analysis with oblique rotation to identify latent dimensions across all items. We found four correlated latent dimensions of AIL, which were mostly interpretable as the abilities to use and interact with AI, to design/program AI (incl. in-depth technical knowledge), to perform complex cognitive operations regarding AI (e.g., ethical considerations), and a common factor for the abilities to detect AI/differentiate between AI and humans and manage persuasive influences of AI (i.e., persuasion literacy). Our findings sort the multitude of AIL instruments and reveal four latent core dimensions of AIL. Thus, they contribute importantly to the conceptual understanding of AIL that has been fragmented so far.