Journal of Educational Measurement最新文献

Cognitive Diagnostic Models (CDMs) provide fine-grained diagnostic feedback, but their central component—the Q-matrix—remains costly and labor-intensive to construct. This study explores the automated generation of Q-matrices using general-purpose AI, including ChatGPT-4o, Gemini-2.5-pro, and Claude-sonnet-4. We evaluated two prompting strategies (all-at-once and one-by-one) across TIMSS 2007, TIMSS 2011, and PISA 2012 mathematics assessments. Results show that AI-generated Q-matrices approximate human baselines with competitive model fitting performance (AIC, BIC, log-likelihood, and SRMSR) and acceptable classification discrepancies. While AI predictions for larger and more complicated assessments (TIMSS 07 and 11) were generally sparser than human-generated Q-matrices, they still achieved equal or better fit statistics under most CDMs. In contrast, for the smaller and less complicated PISA 2012 assessment, AI-generated Q-matrices matched human density and fitting quality. Importantly, chatbot-human matching accuracy remained high across models, with Gemini benefiting from all-at-once prompting, ChatGPT-4o maintaining stable performance under both strategies, and Claude showing sensitivity to prompt structure. These findings highlight both the promise and current limitations of automated Q-matrix generation, underscoring opportunities for integrating LLMs into scalable diagnostic assessment practices.

Fang, L., Lee, G., & Zhai, X. (2025), Using GPT-4 to augment imbalanced data for automatic scoring. Journal of Educational Measurement, 62, 959–995. https://doi.org/10.1111/jedm.70020

Gyeonggeon Lee carried out his work for this article while at the University of Georgia and continued it after moving to Nanyang Technological University, Singapore, where he is now affiliated. The original article erroneously indicated that the work was conducted solely at the University of Georgia.

We apologize for this error.

Unbiasedness for proficiency estimates is important for autoscoring engines since the outcome might be used for future learning or placement. Imbalanced training data may lead to certain biases and lower the prediction accuracy for classification algorithms. In this article, we investigated several data augmentation methods to lower the negative effect of imbalanced data in measurement settings. Four approaches were examined: (1) Resampling methods, either oversampling or undersampling; (2) Active resampling methods, where the resampling weight is based on representativeness in the training set; (3) Data expansion methods using synonym Replacement, slightly changing the meaning or semantics of the original answers; and (4) Content recreation method using Generative AI (e.g., ChatGPT) to create responses for less populated scores. We compared the performance (e.g., Accuracy, QWK, F1) as well as the distance metric for different combinations of the methods. Two datasets with different imbalanced distributions were used. Results show that all four methods can help to mitigate the bias issue and the efficacy was influenced by the imbalance level, representativeness of the original data and the level of increment in the variety of the response (i.e., lexical diversity). In general, resampling and GenAI with active resampling showed the best overall performance.

Advancements in artificial intelligence (AI) have brought significant changes to testing practices, including the emergence of randomly parallel testing (RPT), in which examinees receive different but psychometrically similar sets of items generated from templates or AI-based systems. This paper presents a generalizability theory (GT) framework for estimating conditional standard errors of measurement (CSEMs) and related reliability indices, with a particular focus on design structures commonly encountered in RPT within domain-referenced testing contexts. The proposed framework supports the evaluation of score precision across a variety of operational designs, including crossed, nested, and multivariate configurations. Several illustrative examples are provided to demonstrate the methodology in practical settings. The paper also addresses key psychometric and interpretive challenges associated with RPT and outlines promising directions for future research.

The rapid transition from traditional paper-and-pencil tests to computer-based testing systems has significantly altered the educational landscape, particularly during the COVID-19 pandemic. While online assessments offer numerous advantages, they also present unique challenges, with test security being paramount. This article addresses the critical issue of test fraud in digital assessments, specifically focusing on item preknowledge, where examinees have prior access to test items. Using response-time data, we propose a statistical framework for simultaneously identifying compromised items and examinees with item preknowledge in a single-step analysis. Unlike existing methods, our model does not require prior knowledge about the compromised status of items. Using a large-scale online certification exam dataset, we demonstrate the model's application in detecting significant signals in response times, identifying potentially compromised items, and examinees with potential item preknowledge.

The emergence of automated item generation (AIG) techniques has intensified discussions around their application in assessment development. Some testing companies have already begun developing software to construct exams using AIG. However, the current literature offers limited insights into the characteristics of items generated through AIG, particularly in the realm of multistage testing (MST). This study proposes a novel approach for adjusting template item parameters to enhance ability estimation accuracy under the MST context. A simulation study was conducted using two MST designs with varying numbers of stages and modules. Results demonstrated that the proposed method significantly improved the accuracy of person parameter estimates compared to a more practical, yet less precise, approach that assumes all item clones share identical parameters.

Comparative judgment is an assessment method where item ratings are estimated based on rankings of subsets of the items. These rankings are typically pairwise, with ratings taken to be the estimated parameters from fitting a Bradley-Terry model. Likelihood penalization is often employed to ensure finiteness of estimates. Adaptive scheduling of the comparisons can increase the efficiency of the assessment. We show that the most commonly used penalty in Comparative Judgment is not the best-performing penalty under adaptive scheduling and can lead to substantial bias in parameter estimation. We demonstrate this using simulated and real data and provide a theoretical explanation for the relative performance of the penalties considered, including identifying a preferred alternative. Further, we propose a novel approach based on a parametric bootstrap. It is found to produce better parameter estimates for adaptive schedules and to be robust to variations in underlying strength distributions. The work allows for more efficient implementations of comparative judgment.

The emergence of automated item generation (AIG) techniques has intensified discussions around their application in assessment development. Some testing companies have already begun developing software to construct exams using AIG. However, the current literature offers limited insights into the characteristics of items generated through AIG, particularly in the realm of multistage testing (MST). This study proposes a novel approach for adjusting template item parameters to enhance ability estimation accuracy under the MST context. A simulation study was conducted using two MST designs with varying numbers of stages and modules. Results demonstrated that the proposed method significantly improved the accuracy of person parameter estimates compared to a more practical, yet less precise, approach that assumes all item clones share identical parameters.

Comparative judgment is an assessment method where item ratings are estimated based on rankings of subsets of the items. These rankings are typically pairwise, with ratings taken to be the estimated parameters from fitting a Bradley-Terry model. Likelihood penalization is often employed to ensure finiteness of estimates. Adaptive scheduling of the comparisons can increase the efficiency of the assessment. We show that the most commonly used penalty in Comparative Judgment is not the best-performing penalty under adaptive scheduling and can lead to substantial bias in parameter estimation. We demonstrate this using simulated and real data and provide a theoretical explanation for the relative performance of the penalties considered, including identifying a preferred alternative. Further, we propose a novel approach based on a parametric bootstrap. It is found to produce better parameter estimates for adaptive schedules and to be robust to variations in underlying strength distributions. The work allows for more efficient implementations of comparative judgment.

Aberrant behaviors among test-takers in large-scale assessments are often more prevalent within specific groups or testing sites. While various techniques have been developed to detect individual-level test-takers' aberrant behaviors, research in detecting those behaviors at the group level is rare. We propose a group fit statistic $�l_{�z�2�}$ by extending the $�l_z$ statistic to a multilevel item response model. This new statistic demonstrates adequate power and effectively controls the Type I error rate, particularly when true latent variable values are used or when group sizes are large, such as 500. When latent variable estimates are employed, an adjustment to the $�l_{�z�2�}$ based on the posterior predictive checking approach can offer improved control over the Type I error rate.