Meta-Analysing the Factor Structure and Reliability of Measurement Instruments: An R-Based Tutorial

Abstract

Meta-analytical techniques are crucial to synthetize available evidence and advance research and practice. However, meta-analyses are typically concerned with the strength of an effect and less so with the measurement quality of the scales underlying that effect. This is a problem because measurement quality is crucial for appropriate statistical inferences in studies using multi-item scales. Using 12 samples of the Dirty Dozen questionnaire to assess the dark triad personality, we illustrate (and provide the R scripts for) two methods to meta-analytically assess the factor structure of a scale depending on the data that is available. Further, we provide R scripts for several other methods that we do not discuss in this article in depth. We also illustrate the Reliability Generalization method to meta-analytically assess the reliability of a scale. Strengths and limitations of the different methods are discussed, together with some consideration that researchers must take into account to implement and interpret all methods here. We hope to contribute to the international community of psychologist by equipping researchers with tools that they can use to investigate the measurement quality of different scales, which should enhance the replicability, generalizability and the comparability of research findings.