Comparing the Robustness of Three Nonparametric DIF Procedures to Differential Rapid Guessing

Abstract

ABSTRACT When there are no personal consequences associated with test performance for examinees, rapid guessing (RG) is a concern and can differ between subgroups. To date, the impact of differential RG on item-level measurement invariance has received minimal attention. To that end, a simulation study was conducted to examine the robustness of the Mantel-Haenszel (MH), standardization index (STD), and logistic regression (LR) differential item functioning (DIF) procedures to type I error in the presence of differential RG. Sample size, test difficulty, group impact, and differential RG rates were manipulated. Findings revealed that the LR procedure was completely robust to type I errors, while slightly elevated false positive rates (< 1%) were observed for the MH and STD procedures. An applied analysis examining data from the Programme for International Student Assessment showed minimal differences in DIF classifications when comparing data in which RG responses were unfiltered and filtered. These results suggest that large rates of differences in RG rates between subgroups are unassociated with false positive classifications of DIF.