Journal of Educational Measurement最新文献

Machine learning-based automatic scoring faces challenges with imbalanced student responses across scoring categories. To address this, we introduce a novel text data augmentation framework that leverages GPT-4, a generative large language model specifically tailored for imbalanced datasets in automatic scoring. Our experimental dataset consisted of student-written responses to four science items. We crafted prompts for GPT-4 to generate responses, especially for minority scoring classes, enhancing the dataset. We then fine-tuned DistilBERT for automatic scoring based on the augmented and original datasets. Model performance was assessed using accuracy, precision, recall, and F1 metrics. Our findings revealed that incorporating GPT-4-augmented data significantly improved model performance, particularly in terms of precision and F1 scores. Interestingly, the extent of improvement varied depending on the specific dataset and the proportion of augmented data used. Notably, we found that a varying amount of augmented data (20%-40%) was required to achieve stable improvement in automatic scoring. Comparisons with models trained on additional student-written responses suggest that GPT-4 augmented models align with those trained on student data. This research highlights the potential and effectiveness of data augmentation techniques, utilizing generative large language models like GPT-4, in addressing imbalanced datasets within automatic assessment.

Vertical scales are intended to establish a common metric for scores on test forms targeting different levels of development in a specified domain. They are often constructed using common item, nonequivalent group designs that implicitly rely on the linking items being effectively free from differential item functioning (DIF) or the DIF being symmetric to produce unbiased linking constants. Moderated Nonlinear Factor Analysis (MNLFA) is a measurement model that can be used to understand both the presence of DIF among vertical scale common items and the extent to which the presence of DIF may affect grade-to-grade score distributions. Monte Carlo simulation and synthetic data applications show how models that do and do not account for DIF in vertical scale common items can produce meaningfully different answers to the fundamental question of how much students grow from one grade to the next, but that when DIF is not present, MNLFA provides effectively identical growth estimates to traditional concurrent and characteristic curve approaches to vertical linking.

In U.S. colleges, admissions officers tend to use ACT-SAT concordant scores, also known as linked scores, as predictions of individual scores for tests not taken. The major problem in this situation is the use of linked scores without thoroughly examining their predictive utility (i.e., the degree to which they serve as predicted scores at the individual level). To address this problem, we developed a method, referred to as the “predictive utility analysis,” for quantitatively evaluating the prediction accuracy and error properties of linked scores. A Monte Carlo simulation provided several findings on the behavior of the indices formulated in this paper regarding the number of common examinees, the number of items, and the correlation between tests. Furthermore, we illustrated the predictive utility analysis in concordance and equating with the results of an actual large-scale test, the Japan Law School Admission Test. In both examples, we found that the linked scores obtained by using the equipercentile or linear equating method could be used as predictions of individual scores. Our findings suggest that the predictive utility analysis offers practical guidance for enhancing the use of linked scores as well as supporting institutional accountability.

Automated scoring systems provide multiple benefits but also pose challenges, notably potential bias. Various methods exist to evaluate these algorithms and their outputs for bias. Upon detecting bias, the next logical step is to investigate its cause, often by examining feature distributions. Recently, Johnson and McCaffrey proposed an exploratory approach to identify features responsible for differential prediction bias. However, their approach applies only to linear additive prediction models, excluding many machine learning algorithms. In this paper, we propose the bias contribution measure, a statistic that expands Johnson and McCaffrey's approach to any prediction algorithms that have partial derivatives and that can be implemented in any framework that supports automatic differentiation and matrix inversion. We demonstrated its application and effectiveness on synthetic and real-word data using multiple nonlinear prediction algorithms, including a single-layer feed-forward network (FFN), a support vector regressor, and a deep FFN with multiple hidden layers. In the synthetic data examples, the bias contribution measure successfully identified the feature responsible for the bias. When applied to a real-world data set, the bias contribution measure consistently identified the same set of features across all considered prediction algorithms.

We fine-tuned and compared several encoder-based Transformer large language models (LLM) to predict differential item functioning (DIF) from the item text. We then applied explainable artificial intelligence (XAI) methods to identify specific words associated with the DIF prediction. The data included 42,180 items designed for English language arts and mathematics summative state assessments among students in grades 3 to 11. Prediction $�R^2$ ranged from .04 to .32 among eight focal and reference group pairs. Our findings suggest that many words associated with DIF reflect minor subdomains included in the test blueprint by design, rather than construct-irrelevant content that may need to be removed from assessments. This may explain why qualitative reviews of DIF items often yield inconclusive results. Our approach can be used to (1) screen words associated with DIF during the item-writing process for immediate revision to reduce preventable adverse DIF, (2) assist traditional DIF item reviews by highlighting key words, or (3) use DIF prediction as an alternative when obtaining sufficient sample size for traditional DIF analyses is impossible. Extensions of this research can enhance the assessment fairness, especially programs that lack resources to build high-quality items, and among smaller subpopulations with insufficient sample sizes for traditional DIF analyses. See source code here.

This study builds on prior research on adaptive testing by examining the performance of item calibration methods in the context of multidimensional multistage tests with within-item multidimensionality. Building on the adaptive module-level approach, where test-takers proceed through customized modules based on their initial performance, this research investigates how different calibration methods perform under certain conditions. Specifically, the study evaluates three calibration methods—concurrent calibration, fixed item parameter calibration, and concurrent calibration with multiple panels—within a multidimensional multistage test framework. Using computer simulations, the study assesses ability and item parameter recovery across various conditions, including sample size, correlations among dimensions, and routing stage length. Across 36 simulation conditions, each replicated 10 times, results show that although calibration methods exert minimal influence on item and ability parameter estimates, the correlation among dimensions plays a significant role in both item and ability estimation. Additionally, sample size and routing stage length notably impact the estimation of item discrimination parameters. This study lays the foundation for further research and practical advancements in multidimensional multistage testing, offering a starting point for refining and innovating testing practices.

Collaborative problem solving is widely recognized as a critical 21st-century skill. Assessing collaborative problem solving depends on coding the communication data using a construct-relevant framework, and this process has long been a major bottleneck to scaling up such assessments. Based on five datasets and two coding frameworks, we demonstrate that ChatGPT can code communication data to a satisfactory level, though performance varies across ChatGPT models and depends on the coding framework and task characteristics. Interestingly, newer reasoning-focused models, such as GPT-o1-mini and GPT-o3-mini, do not necessarily yield better coding results. Additionally, we show that refining prompts based on feedback from miscoded cases can improve coding accuracy in some instances, though the effectiveness of this approach is not consistent across all tasks. These findings offer practical guidance for researchers and practitioners in developing scalable, efficient methods to analyze communication data in support of 21st-century skill assessment.

Use of artificial intelligence (AI) to score responses is growing in popularity and likely to increase. Evidence of the validity of scores relies on quadratic weighted kappa (QWK) to demonstrate agreement between AI scores and human ratings. QWK is a measure of agreement that accounts for chance agreement and the ordinality of the data by giving greater weight to larger disagreements. It has known shortcomings including sensitivity to the human rating reliability. The proportional reduction in mean squared error (PRMSE) measures agreement between predictions and their target that accounts for measurement error in the target. For example, the accuracy of the automated scoring model, with respect to prediction of the human true scores rather than the observed ratings. Extensive simulation study results show PRMSE is robust to many factors to which QWK is sensitive such as the human rater reliability, skew in the data and the number of score points. Analysis of operational test data demonstrates QWK and PRMSE can lead to different conclusions about AI scores. We investigate sample size requirements for accurate estimation of PRMSE in the context of AI scoring, although the results could apply more generally to measures with similar distributions as those tested in our study.

Online calibration estimates new item parameters alongside previously calibrated items, supporting efficient item replenishment. However, most existing online calibration procedures for Cognitive Diagnostic Computerized Adaptive Testing (CD-CAT) lack mechanisms to ensure content balance during live testing. This limitation can lead to uneven content coverage, potentially undermining the alignment with instructional goals. This research extends the current calibration framework by integrating a two-phase test design with a content-balancing item selection method into the online calibration procedure. Simulation studies evaluated item parameter recovery and attribute profile estimation accuracy under the proposed procedure. Results indicated that the developed procedure yielded more accurate new item parameter estimates. The procedure also maintained content representativeness under both balanced and unbalanced constraints. Attribute profile estimation was sensitive to item parameter values. Accuracy declined when items had larger parameter values. Calibration improved with larger sample sizes and smaller parameter values. Longer test lengths contributed more to profile estimation than to new item calibration. These findings highlight design trade-offs in adaptive item replenishment and suggest new directions for hybrid calibration methods.

This study considers the estimation of marginal reliability and conditional accuracy measures using a generalized recursion procedure with several IRT-based ability and score estimators. The estimators include MLE, TCC, and EAP abilities, and corresponding test scores obtained with different weightings of the item scores. We consider reliability estimates for 1-, 2-, and 3-parameter logistic IRT models (1PL, 2PL, and 3PL) for tests of dichotomously scored items, using IRT calibrations from two datasets. The generalized recursion procedure is shown to produce conditional probability distributions for the considered IRT estimators that can be used in the estimation of marginal reliabilities and conditional accuracies (biases and CSEMs). These reliabilities and conditional accuracies are shown to have less extreme and more plausible values compared to theoretical approaches based on test information. The proposed recursion procedure for the estimation of reliability and other accuracy measures are demonstrated for testing situations involving different test lengths, IRT models, and different types of IRT parameter inaccuracies.