Educational Measurement-Issues and Practice最新文献

This case study applied the weak theory of Automatic Item Generation (AIG) to generate isomorphic item instances (i.e., unique but psychometrically equivalent items) for a large-scale assessment. Three representative instances were selected from each item template (i.e., model) and pilot-tested. In addition, a new analytical framework, differential child item functioning (DCIF) analysis, based on the existing differential item functioning statistics, was applied to evaluate the psychometric equivalency of item instances within each template. The results showed that, out of 23 templates, nine successfully generated isomorphic instances, five required minor revisions to make them isomorphic, and the remaining templates required major modifications. The results and insights obtained from the AIG template development procedure may help item writers and psychometricians effectively develop and manage the templates that generate isomorphic instances.

In addition to an exciting new module on Multidimensional Item Response Theory (MIRT) equating, there are two important announcements regarding the Instructional Topics in Educational Measurement Series (ITEMS). After much discussion with authors, learners, the educational measurement community, and other stakeholders of ITEMS, I am pleased to announce (1) the transfer of the ITEMS portal to the National Council on Measurement in Education (NCME) website and (2) a new digital module format.

Transfer of the ITEMS portal to the NCME website: In 2018, I, along with Matthew Gaertner, led efforts to launch the new NCME website on the Higher Logic platform. Besides bringing a modern look and feel to the organization's web presence, the platform was selected due to its flexibility and customizability. In the years since, traffic to the NCME website has continued to increase, and there has been a significant increase in site content (e.g., software database, special interest group community pages). In April of this year, just prior to the NCME Annual Conference, the website committee, led by Erin Banjanovic, released a much-needed re-organization of the content on the site. This wonderful overhaul has made navigating the NCME website easier with content now in more logical locations. However, noticeably absent from the NCME website has been the ITEMS portal. As a reminder, ITEMS is a publication from NCME that has a brief summary published in the Educational Measurement: Issues and Practice journal with the primary digital content on the ITEMS portal, freely available after registration. The ITEMS portal has been a Learning Management System based website, with many features and ripe for extension. Though from the user's perspective, it can be complex to navigate and necessitates additional navigation from the primary NCME website, also requiring unique log-in criteria.

It is at this time that I am pleased to announce that the ITEMS portal is now available on the NCME website at the following link: https://www.ncme.org/itemsportal

Transferring the ITEMS portal to the NCME website has several immediate benefits. First, all modules will remain free of charge, but no longer require additional registration. Second, they will have a different organization structure, improving navigating across modules and enabling more efficient access to key information. Finally, they will fall under the NCME brand, having the same look and feel with all the content on the NCME website.

Although this issue marks the launch of the ITEMS portal on the NCME website, the transfer of content remains a work in progress. For now, both the old and new ITEMS portal will be available and all links to the old ITEMS portal will remain functional. However, I would strongly advise all who embed or link to content to begin updating to the portal on the NCME website. Nearly all of the content has been shifted, but if you notice anything missing or ha

The focus of this paper is on the empirical relationship between item difficulty and item discrimination. Two studies—an empirical investigation and a simulation study—were conducted to examine the association between item difficulty and item discrimination under classical test theory and item response theory (IRT), and the effects of the association on various quantities of interest. Results from the empirical investigation show that item difficulty and item discrimination are negatively correlated under classical test theory, mostly negatively correlated under the two-parameter logistic model, and mostly positively correlated under the three-parameter logistic model; the magnitude of the correlation varied over the different data sets. Results from the simulation study reveal that a failure to incorporate the correlation between item difficulty and item discrimination in IRT simulations may provide the investigator with inaccurate values of important quantities of interest, and may lead to incorrect operational decisions. Implications to practice and future directions are discussed.

In this digital ITEMS module, Dr. Stella Kim provides an overview of multidimensional item response theory (MIRT) equating. Traditional unidimensional item response theory (IRT) equating methods impose the sometimes untenable restriction on data that only a single ability is assessed. This module discusses potential sources of multidimensionality and presents potential consequences of multidimensionality on equating. To remedy these effects, MIRT equating can be used as a viable alternative to traditional methods of IRT equating. In conducting MIRT equating, the choice of an appropriate MIRT model is necessary, and thus the module describes several existing MIRT models and illustrates each using hypothetical examples. After a brief description of MIRT models, an extensive review of the current literature is presented to identify gaps in the literature on MIRT equating. Then, the steps for conducting MIRT observed-score equating are described. Finally, the module discusses practical considerations in applying MIRT equating to testing practices and suggests potential areas of research for future studies.

Many scholars compared various item discrimination indices in real or simulated data. Item discrimination indices, such as item-total correlation, item-rest correlation, and IRT item discrimination parameter, provide information about individual differences among all participants. However, there are tests that aim to select a very limited number of students, examinees, or candidates for allocated schools and job positions. Thus, there is a need to evaluate the performances of CTT and IRT item discrimination indices when the test purpose is to select a limited number of students. The purpose of the current Monte Carlo study is to evaluate item discrimination indices in the case of selecting a limited number of high-achieving students. The results showed that a special case of Brennan's index, B_10–90, provided more accurate information for this specific test purpose. Additionally, the effects of various factors, such as test length, ability distributions of examinees, and item difficulty variance on item discrimination indices were investigated. The performance of each item discrimination index is discussed in detail.

We address two persistent challenges in large-scale assessments of the Next Generation Science Standards: (a) the validity of score interpretations that target the standards broadly and (b) how to structure claims for assessments of this complex domain. The NGSS pose a particular challenge for specifying claims about students that evidence from summative assessments can support. As a solution, we propose tiered claims, which explicitly distinguish between claims about what students have done or can do on test items—which are typically easier to support under current test designs—and claims about what students could do in the broader domain of performances described by the standards, for which novel evidence is likely required. We discuss the positive implications of tiered claims for test construction, validation, and reporting of results.

I show that there are better measures of student college performance than grade point average (GPA) by undertaking a fine-grained empirical investigation of grading within a large public university. The value of using GPA as a measure of comparative performance is undermined by academically weaker students taking courses where the grading is more generous. In fact, college courses composed of weaker performing students (whether measured by their relative performance in other classes, SAT scores, or high school GPA) have higher average grades. To partially correct for idiosyncratic grading across classes, alternative measures, student class rank and the student's average class rank, are introduced. In comparison to a student's lower-division grade, the student's lower-division rank is a better predictor of the student's grade in the upper-division course. Course rank and course grade are adjusted to account for different levels of academic competitiveness across courses (more precisely, student fixed-effects are derived). SAT scores and high school GPA are then used to predict college performance. Higher explained variation (R²) is obtained when the dependent variable is average class rank rather than GPA. Still higher explained variation occurs when the dependent variable is adjusted rank.

Coefficient alpha reliability persists as the most common reliability coefficient reported in research. The assumptions for its use are, however, not well-understood. The current paper challenges the commonly used expressions of coefficient alpha and argues that while these expressions are correct when estimating reliability for summed scores, they are not appropriate to extend coefficient alpha to correctly estimate the reliability for nonlinearly transformed scaled scores such as percentile ranks and stanines. The current paper reconceptualizes coefficient alpha as a complement of the ratio of two unbiased estimates of the summed score variance. These include conditional summed score variance assuming uncorrelated item scores (gives the error score variance) and unconditional summed score variance incorporating intercorrelated item scores (gives the observed score variance). Using this reconceptualization, a new equation of coefficient generalized alpha is introduced for scaled scores. Coefficient alpha is a special case of this new equation since the latter reduces to coefficinet alpha if the scaled scores are the summed scores themselves. Two applications (cognitive and psychological assessments) are used to compare the performance (estimation and bootstrap confidence interval) of the reliability coefficients for different scaled scores. Results support the new equation of coefficient generalized alpha and compare it to coefficient generalized beta for parallel test forms. Coefficient generalized alpha produced different reliability values, which were larger than coefficient generalized beta for different scaled scores.