Journal of Educational Measurement最新文献

Prompts play a crucial role in eliciting accurate outputs from large language models (LLMs). This study examines the effectiveness of an automatic prompt engineering (APE) framework for automatic scoring in educational measurement. We collected constructed-response data from 930 students across 11 items and used human scores as the true labels. A baseline was established by providing LLMs with the original human-scoring instructions and materials. APE was then applied to optimize prompts for each item. We found that on average, APE increased scoring accuracy by 9%; few-shot learning (i.e., giving multiple labeled examples related to the goal) increased APE performance by 2%; a high temperature (i.e., a parameter for output randomness) was needed in at least part of the APE to improve the scoring accuracy; Quadratic Weighted Kappa (QWK) showed a similar pattern. These findings support the use of APE in automatic scoring. Moreover, compared with the manual scoring instructions, APE tended to restate and reformat the scoring prompts, which could give rise to concerns about validity. Thus, the creative variability introduced by LLMs raises considerations about the balance between innovation and adherence to scoring rubrics.

Constructed responses (CRs) within testlets are widely used to assess complex skills but can pose calibration challenges due to local item dependence. A few current testlet models incorporate testlet-specific effects to address local dependence but struggle with interpreting these effects and may not fully capture the complexities of CR items because they rely only on response or score patterns. A Topic Testlet Model (TTM) integrates topic modeling within a psychometric framework was proposed. It uses latent topics from student written responses to adjust for local dependence, enable simultaneous calibration, and provide insights into evaluating student reasoning and writing in testlet CR items. Using empirical data from both English Language and Arts as well as Science assessments for grades 3-12, we compare the TTM with existing models in terms of ability estimates, item parameter estimates, and overall model fit. Simulation studies further demonstrate parameter recovery under various testing scenarios. Results show that the TTM effectively accounts for local dependence, improves testlet effect interpretability, and demonstrates a better fit than the existing models. TTM advances CR testlet calibration, leveraging additional information from student written responses to improve the precision of the assessment systems and validity of the use of test scores.

Large-scale survey assessments (LSAs) such as NAEP, TIMSS, PIRLS, IELS, and NAPLAN produce plausible values of student proficiency for estimating population statistics. Plausible values are imputed values for latent proficiency variables. While prominently used for LSAs, they are applicable to a wide range of latent variable modelling contexts such as surveys about psychological dispositions or beliefs. Following the practice of multiple imputation, LSAs produce multiple sets of plausible values for each survey. The criteria used to determine the number of plausible values remains unresolved and is inconsistent in practice. We show analytically and via simulation that the number of plausible values used determines the amount of Monte Carlo error on point estimates and standard errors as a function of the fraction of missing information. We derive expressions to determine the number of plausible values required to reach a given level of precision. We analyze real data from a LSA to provide guidelines supported by theory, simulation, and real data on the number of plausible values. Finally, we illustrate the impact with a power analysis. Our results show there is meaningful benefit to the use of greater numbers of plausible values than currently generated by LSAs.

While testlets have proven useful for assessing complex skills, the stem shared by multiple items often induces correlations between responses, leading to violations of local independence (LI), which can result in biased parameter and ability estimates. Diagnostic procedures for detecting testlet effects typically involve model comparisons testing for the inclusion of extra testlet parameters or, at the item level, testing for pairwise LI. Rosenbaum's adaptation of the Mantel-Haenszel (MH) $�{\chi }^2$-statistic belongs to the latter category. The MH $�{\chi }^2$-statistic has also been used in cognitive diagnosis for detecting violations of LI and for the identification of testlet effects. However, this approach is not without limitations, as it lacks a rationale for integrating multiple pairwise MH $�{\chi }^2$-statistics and any notion of the sampling distribution of such an integrated statistic. In this article, a procedure for integrating multiple pairwise MH $�{\chi }^2$-statistics to evaluate testlet effects in cognitive diagnosis is proposed. The unknown sampling distribution issue is addressed by implementing a parametric bootstrap resampling scheme. Results from simulation studies demonstrate the performance of the proposed parametric bootstrap testlet MH $�{\chi }^2$-statistic, and its application to the 2015 PISA Collaborative Problem Solving (CPS) data set illustrates the method's practical merits.

Alternate assessments of the same construct or assessments that have undergone a change in the conditions of measurement are often linked in an attempt to establish score comparability. As the link must be estimated from the data, linking contributes error variance into estimators. We propose a novel method to account for linking variance in standard error estimation for achievement or proficiency levels, a primary outcome for many international, national, and U.S. state assessments. Achievement levels are proportions of a population within some range of ability, such as the proportion of the population classified as proficient or advanced. The method is validated in a simulation and with real data. Our method allows for sampling weights and complex sampling and involves an easily calculated correction term that may be added to conventional estimates of the error variance, correcting the conventional estimates for neglecting variance due to linking. Furthermore, the method accounts for dependencies between linking with other sources of variance, allowing for the method to be much more widely applicable to a range of score comparisons than traditional methods of linking variance estimation.

In recent years, the application of explainability techniques to automated essay scoring and automated short-answer grading (ASAG) models, particularly those based on transformer architectures, has gained significant attention. However, the reliability and consistency of these techniques remain underexplored. This study systematically investigates the use of attribution scores in ASAG systems, focusing on their consistency in reflecting model decisions. Specifically, we examined how attribution scores generated by different methods—namely Local Interpretable Model-agnostic Explanations (LIME), Integrated Gradients (IG), Hierarchical Explanation via Divisive Generation (HEDGE), and Leave-One-Out (LOO)—compare in their consistency and ability to illustrate the decision-making processes of transformer-based scoring systems trained on a publicly available response dataset. Additionally, we analyzed how attribution scores varied across different scoring categories in a polytomously scored response dataset and across two transformer-based scoring model architectures: Bidirectional Encoder Representations from Transformers (BERT) and Decoding-enhanced BERT with Disentangled Attention (DeBERTa-v2). Our findings highlight the challenges in evaluating explainability metrics, with important implications for both high-stakes and formative assessment contexts. This study contributes to the development of more reliable and transparent ASAG systems.

Response styles pose great threats to psychological measurements. This research compares IRTree models and anchoring vignettes in addressing response styles and estimating the target traits. It also explores the potential of combining them at the item level and total-score level (ratios of extreme and middle responses to vignettes). Four models were evaluated: three multidimensional IRTree models with different levels of using vignette data and a nominal response model (NRM) addressing extreme and midpoint response styles with item-level vignette responses. Simulation results indicated that the IRTree model using item-level vignette responses outperformed others in estimating the target trait and response styles to different extents, with performance improving as the number of vignettes increased. Empirical findings further demonstrated that models using item-level vignette information yielded higher reliability and closely aligned target trait estimates. These results underscore the value of integrating anchoring vignettes with IRTree models to enhance estimation accuracy and control for response styles.

Personalized assessments are of increasing interest because of their potential to lead to more equitable decisions about the examinees. However, one obstacle to the widespread use of personalized assessments is the lack of a measurement toolkit that can be used to analyze data from these assessments. This article takes one step toward building such a toolkit by proposing a validation framework for personalized assessments. The framework is built on the threats-to-validity approach. We demonstrate applications of the suggested framework using the AP 3D Art and Design Portfolio examination and a more restrictive culturally relevant assessment as examples.

This article addresses bias in Spoken Language Systems (SLS) that involve both Automatic Speech Recognition (ASR) and Natural Language Processing (NLP) and reports experiments to improve the performance of SLS for automated language and literacy-related assessments with students who are under served in the U.S. educational system. We frame bias in SLS in terms of testing fairness and validity, stemming in part from the exclusion of sufficiently large training datasets in varieties of English other than General American English (GAE). We adopt an Interpretation/Use Argument approach to validity focused on clarity of constructs and scoring accuracy. While SLS use ASR to automatically transcribe students’ utterances, and apply NLP algorithms to ASR transcripts to measure students’ speech samples, it is well-documented in studies with adults that ASR is typically more problematic for African American English (AAE) speakers than for other groups due to differences in prosody, pronunciation, word usage, and grammar. We utilized child speech and text corpora to improve algorithms that score oral task responses for child AAE speakers and, in some experiments, children with oral language and reading difficulties. Favorable results provide impetus and possible solutions for fair and inclusive assessments for diverse student groups in the future.

In computerized adaptive testing, item exposure control methods are often used to provide a more balanced usage of the item pool. Many of the most popular methods, including the restricted method (Revuelta and Ponsoda), use a single maximum exposure rate to limit the proportion of times that each item is administered. However, Barrada et al. showed that by using multiple maximum exposure rates, it is possible to obtain an even more balanced usage of the item pool. Therefore, in this paper, we develop four extensions of the restricted method that involve the use of multiple maximum exposure rates. A detailed simulation study reveals that (a) all four of the new methods improve item pool utilization and (b) three of the new methods also improve measurement accuracy. Taken together, these results are highly encouraging, as they reveal that it is possible to improve both types of outcomes simultaneously.