Examining Validity Evidence of Self-Report Measures Using Differential Item Functioning: An Illustration of Three Methods

Abstract

The investigation of differential item functioning (DIF) is important for any group comparison because the validity of the inferences made from scale scores could be compromised if DIF is present. DIF occurs when individuals from different groups show different probabilities of selecting a response option to an item after being matched on the underlying latent variable that the item is supposed to measure. The aim of this paper is to inform the practice of DIF analyses in survey research. We focus on three quantitative methods to detect DIF, namely nonparametric item response theory (NIRT), ordinal logistic regression (OLR), and mixed-effects or multilevel models. Using these methods, we demonstrate how to examine DIF at the item and scale levels, as well as in multilevel settings. We discuss when these techniques are appropriate to use, what data assumptions they have, and their advantages and disadvantages in the analysis of survey data.