Item-Fit Statistic Based on Posterior Probabilities of Membership in Ability Groups.

Abstract

A novel approach to item-fit analysis based on an asymptotic test is proposed. The new test statistic, ${\chi }_w^2$ , compares pseudo-observed and expected item mean scores over a set of ability bins. The item mean scores are computed as weighted means with weights based on test-takers' a posteriori density of ability within the bin. This article explores the properties of ${\chi }_w^2$ in case of dichotomously scored items for unidimensional IRT models. Monte Carlo experiments were conducted to analyze the performance of ${\chi }_w^2$ . Type I error of ${\chi }_w^2$ was acceptably close to the nominal level and it had greater power than Orlando and Thissen's $S-x^2$ . Under some conditions, power of ${\chi }_w^2$ also exceeded the one reported for the computationally more demanding Stone's ${\chi }^{2\ast }$ .