Educational Measurement-Issues and Practice最新文献

This article is based on my 2023 NCME Presidential Address, where I talked a bit about my journey into the profession, and more substantively about comparable scores. Specifically, I discussed some of the different ways ‘comparable scores’ are defined, highlighted some areas I think we as a profession need to pay more attention to when considering score comparability, and emphasized that comparability in this context is a matter of degree which varies according to the decisions we plan to make on particular scores.

The remarkable strides in artificial intelligence (AI), exemplified by ChatGPT, have unveiled a wealth of opportunities and challenges in assessment. Applying cutting-edge large language models (LLMs) and generative AI to assessment holds great promise in boosting efficiency, mitigating bias, and facilitating customized evaluations. Conversely, these innovations raise significant concerns regarding validity, reliability, transparency, fairness, equity, and test security, necessitating careful thinking when applying them in assessments. In this article, we discuss the impacts and implications of LLMs and generative AI on critical dimensions of assessment with example use cases and call for a community effort to equip assessment professionals with the needed AI literacy to harness the potential effectively.

Short scales are time-efficient for participants and cost-effective in research. However, researchers often mistakenly expect short scales to have the same reliability as long ones without considering the effect of scale length. We argue that applying a universal benchmark for alpha is problematic as the impact of low-quality items is greater on shorter scales. In this study, we proposed simple guidelines for item reduction using the “alpha-if-item-deleted” procedure in scale construction. An item can be removed if alpha increases or decreases by less than .02, especially for short scales. Conversely, an item should be retained if alpha decreases by more than .04 upon its removal. For reliability benchmarks, .80 is relatively safe in most conditions, but higher benchmarks are recommended for longer scales and smaller sample sizes. Supplementary analyses, including item content, face validity, and content coverage, are critical to ensure scale quality.

Measuring opportunities to learn (OTL) is crucial for evaluating education quality and equity, but obtaining accurate and comprehensive OTL data at a large scale remains challenging. We attempt to address this issue by investigating measurement concerns in data collection and sampling. With the primary goal of estimating group-level OTLs for large populations of classrooms and the secondary goal of estimating classroom-level OTLs, we propose forming a teacher panel and using an online log-type survey to collect content and time data on sampled days throughout the school year. We compared various sampling schemes in a simulation study with real daily log data from 66 fourth-grade math teachers. The findings from this study indicate that sampling 1 day per week or 1 day every other week provided accurate group-level estimates, while sampling 1 day per week yielded satisfactory classroom-level estimates. The proposed approach aids in effectively monitoring large-scale classroom OTL.

This issue marks the beginning of the final year of my tenure as editor of the Instructional Topics of Educational Measurement Series (ITEMS). Although I will save a comprehensive reflection until the last issue of the year, I will use this issue to provide an update on the two changes to ITEMS that were made over the past two years in addition to introducing the newest entry to the ITEMS digital library.

In 2022, I took over a newly created format and process for ITEMS modules—digital interactive teaching modules as opposed to the traditional PDF format. Modules were hosted on a learning management system (LMS), shifting focus from a journal publication to a modernized teaching-focused platform. While the LMS provided ample opportunity for expansion and learning, its form was challenging to navigate given its unique look and feel as a standalone website, separate from the then recently revitalized modern NCME website. Thus, my first significant change to ITEMS was to migrate the ITEMS portal LMS to the NCME website. Learners now go straight to the NCME website and navigate directly to the ITEMS portal without requiring a unique username and password. This makes the ITEMS module library more accessible and makes the NCME website the go-to destination for professional development in educational measurement.

Prior to 2022, specialized software was necessary to produce digital modules. This software allowed for the inclusion of interactive and nonlinear learning. This process resulted in beautiful, intricate, and pleasing modules for learning, but was complex and costly for development. To continue the process in place would have required producing these modules as a full-time job with a staff—a true testament to the work of the previous editor André Rupp and other volunteers. Fortunately, the ITEMS portal being hosted on the NCME website offered an opportunity to modify the development process while keeping an interactive component with a modernized look and feel.

I developed a new comprehensive system for creating ITEMS modules. Authors now develop content for four to five sections using pre-made PowerPoint templates, a familiar software to many. Once completed, I insert animations to align with timing of the recorded audio and export them as videos. These videos are hosted on the NCME website and can be downloaded for offline viewing (an extra benefit). Learners interact directly with the videos and content on the website, where module sections may be viewed in any order. In addition to video content, interactive activities that exemplify syntax or case studies are developed to assist with learning. Finally, interactive selected response learning checks are made, allowing learners to check their understanding as they traverse through the module. Datasets, syntax, and other files are hosted on the module page for easy download.

The ITEMS development process is not l