ETS Research Report Series最新文献

Substantial progress has been made toward applying technology enhanced conversation-based assessments (CBAs) to measure the English-language proficiency of English learners (ELs). CBAs are conversation-based systems that use conversations among computer-animated agents and a test taker. We expanded the design and capability of prior conversation-based instructional and assessment systems and developed a CBA designed to measure the English language skills and the mathematics knowledge of middle school ELs. The prototype CBA simulates an authentic and engaging mathematics classroom where the test taker interacts with two virtual agents to solve math problems. We embedded feedback and supports that are triggered by how the CBA interprets students' written responses. In this study, we administered the CBA to middle school ELs (N = 82) residing in the United States. We examined the extent to which the CBA system was able to consistently interpret the students' responses (722 responses for the 82 students). The study findings helped us to understand the factors that affect the accuracy of the CBA system's interpretations and shed light on how to improve CBA systems that incorporate scaffolding.

In educational assessment, low engagement is problematic when tests are low stakes for students but have significant consequences for teachers or schools. In the current study, we sought to establish the electroencephalographic (EEG) correlates of engagement and to distinguish engagement from mental effort. Forty university students participated in a simulated GRE® General Test session while scalp EEG was recorded from 128 channels. Participants completed two verbal and two quantitative GRE test blocks for a total of 40 items each, and after each item, rated either their engagement or mental effort on a scale of 1–6. We computed power for seven frequency bands (delta, theta, alpha, beta, and low, medium, and high gamma) across six regions of interest: left hemisphere (LH) and right hemisphere (RH) frontal, temporal, and parietal. Preliminary results suggested that gamma power (30–150 hertz [Hz]) indexed differences between high- and low-engagement ratings. This pattern was similar but weaker for mental effort. A cumulative logit model with cross-classified random effects determined that high gamma (90–150 Hz) over the LH temporal cortex predicted engagement ratings, while controlling for reaction time and accuracy. However, for effort ratings, reaction time was the sole significant predictor. These results suggest that high gamma may be a correlate of engagement during complex cognitive tasks, but not a correlate of effort. The findings are a promising step toward the goal of objectively measuring engagement during assessment tasks.

This study evaluated the impact of subgroup weighting for equating through a common-item anchor. We used data from a single test form to create two research forms for which the equating relationship was known. The results showed that equating was most accurate when the new form and reference form samples were weighted to be similar to the target population. When the target population was a combination of the two equating samples and one sample was weighted to be similar to the other, the equating was less accurate but still much more accurate than equating with unweighted samples.