Applying a Multiple Comparison Control to IRT Item-fit Testing

ABSTRACT We used simulation techniques to assess the item-level and familywise Type I error control and power of an IRT item-fit statistic, the S-X2 . Previous research indicated that the S-X2 has good Type I error control and decent power, but no previous research examined familywise Type I error control. We varied percentage of misfitting items, sample size, and test length, and computed familywise Type I error with no correction, a Bonferroni correction, and a Benjamini-Hochberg correction. The S-X2 controlled item-level and familywise Type I errors when corrections were applied to conditions with no misfitting items. In the presence of misfitting items, the S-X2 exhibited inflated item-level and familywise false hit rates in many conditions, even with familywise Type I error corrections. Lastly, power was low and negatively impacted when either of the familywise Type I error corrections was applied. We suggest using the S-X2 with no familywise Type I error control in conjunction with other methods of assessing item fit (e.g., visual analysis).