ETS Research Report Series最新文献

Though a substantial amount of research exists on imputing missing scores in educational assessments, there is little research on cases where responses or scores to an item are missing for all test takers. In this paper, we tackled the problem of imputing missing scores for tests for which the responses to an item are missing for all test takers. We considered three missing-data imputation methods—the median method, the item response theory (IRT) method, and the two-way method—for imputing scores. We compared the performance of these three imputation methods with respect to their accuracy in estimating scaled scores and test reliability for the aforementioned problem. Real data were used in the comparison. All three methods performed well in imputing scaled scores with negligible imputation error: The IRT method and the median method provided slightly more accurate scaled scores. The two-way method provided the most accurate reliability estimates. Recommendations for practice are provided.

This study explores the extent to which student, family, peer, and school factors predict (a) whether students take Advanced Placement® (AP®) courses, International Baccalaureate (IB) courses, and dual enrollment courses and (b) in models limited to course takers, how many courses they completed. Our findings, based on a nationally representative, longitudinal sample, suggest that, when it comes to college-level high school course taking, the relative advantage of higher socioeconomic status (SES) is less for African American students than it is for White and Asian students. Ninth-grade math skills are the strongest predictor of AP or IB and dual enrollment course taking, above and beyond demographic background characteristics like SES and race or ethnicity. High school girls take AP/IB and dual enrollment courses at a higher rate than boys, and they take more of these courses. The level of academic focus of students and their peers is associated with both AP or IB and dual enrollment course taking, whereas having parents focused on college preparation and course taking only predicts AP or IB course taking. School factors associated with AP or IB course taking include U.S. region and rural location; the percentage of math teachers with a master's degree is also positively associated with the number of AP or IB courses students take. These findings highlight the importance of equitable educational opportunities starting from a young age. They also indicate a need for increased early attention to student math skills and for more supports for parents and school staff to enable them to encourage and prepare all students, especially those from historically marginalized groups, to take college-level courses in high school.

Linear functional relationships are intended to be symmetric and therefore cannot generally be accurately estimated using ordinary least squares regression equations. Orthogonal regression (OR) models allow for errors in both Y and X and therefore can provide symmetric estimates of these relationships. The most well-established OR model, the errors-in-variables (EIV) model, assumes that the observed scatter around the line is due entirely to errors of measurement in Y and X and that the ratio of the error variances is known. If most of the variance around the line is known to be due to the errors of measurement in Y and X, the EIV model can provide an unbiased maximum likelihood estimate for a functional relationship. However, if a substantial part of the variability around the line is due to natural variability, which is not attributable to errors of measurement in Y or X, the ratio of the measurement error variances is not well defined and the EIV model is not directly applicable. The main contribution of this report is the development of a hybrid model that provides plausible estimates for linear functional relationships in cases with substantial natural variability and substantial errors of measurement. An analysis of female and male differential test functioning between an essay test and an objective test used as parts of a licensure examination provides an illustration of the use of the hybrid model.

Accomplishing a communication task in the real world requires the ability not only to do the task per se but also to manage aspects of the context in which it occurs. For this reason, simulations of target language use contexts have been incorporated into the design of communicative language tests as a way of enhancing the authenticity of assessment task performance. Although some contextual factors may increase extraneous cognitive load and distract learners from focusing on the task at hand (Sweller, 1994), they represent important design considerations in task-based language assessment (TBLA), where the purpose of assessment is to determine what second language (L2) learners can do with the target language in the real world. In that sense, the extraneous cognitive load might well be part of the construct we are interested in assessing. Accordingly, the current study simulated aspects of a real-world task performance context as part of an email writing task assessment. Simulated context was operationalized as (a) additional information about the task scenario, (b) a visual image to simulate the physical context, and (c) an audio to replicate the real-world experience. A total of 276 L2 English learners performed the email task, half with simulated context and the other half without it. Findings revealed that, when presented with simulated context, the tasks were perceived by participants to have induced more time pressure and to be more interesting. In terms of performance effects, the provision of simulated context negatively affected the syntactic complexity of participants' writing but positively affected their syntactic fluency. It also led to greater discrimination among learners at different proficiency levels on various measures of language performance. The paper concludes by highlighting implications for task design and validity evaluation, especially in TBLA.

Individuals with a high school education represent the largest subset of the U.S. workforce. However, little is known about the employer expectations of these individuals, particularly in the area of soft skills—also known as 21st-century skills. Online job advertisements offer useful data for examining these expectations, as they may supplement employer survey data and reflect actual recruitment practices. Our analysis of 68,505 online job advertisements suggests that employers hold generally lower expectations for the soft skills of high school–educated individuals than they do for postsecondary-educated individuals. However, employer expectations for two soft skills—professionalism and customer service skills—appear to be substantially higher for high school–educated individuals than for postsecondary-educated individuals. Additional results highlight similarities and differences within the high school–educated workforce across nine workplace industries. We discuss the implications of these results not only for high school–educated individuals and the organizations that employ them but also for practitioners and educators charged with assessing and providing training for these skills.

Researchers suggest that claims about the meaningfulness of test score interpretations and consequences of test use should be backed by evidence that stakeholders understand the definition of the construct assessed (meaningfulness) and score reports (consequences). Evaluation of stakeholders' actual uses and interpretations of score reports in large-scale standardized language proficiency tests, however, remains limited in the score reporting literature. This study investigates how test takers, as an important stakeholder group, use and interpret the score report of the TOEIC® Listening and Reading (TOEIC L&R) test. Data were collected from 834 TOEIC L&R test takers in Japan, who represented a wide range of English language proficiency based on their TOEIC L&R total scores. The participants responded to an online survey that asked about their uses and interpretations of the test results and their comprehension of the performance feedback presented in the score report. The results showed that the participants used the test results largely as intended, providing an important piece of validity evidence to support the proposed uses of the TOEIC L&R test. Some of the score reporting elements such as the performance feedback and footnote message, however, were not easy to understand for all participants, revealing a need to improve the interpretability of the score report. The study findings have implications for designing informative score reports and the usefulness around reporting test performance feedback.

In a targeted double-scoring procedure for performance assessments that are used for licensure and certification purposes, a subset of responses receives an independent second rating if the first rating falls into a preidentified critical score range (CSR) where an additional rating would lead to considerably more reliable pass-fail decisions. This study evaluates the CSRs using two approaches—one based on imputation of missing scores and the other based on statistical decision theory—using data from the Performance Assessment for School Leaders (PASL). Results from the evaluation indicate that the currently used CSRs are effective.

This study examined how high school course-taking patterns (i.e., career and technical education [CTE] vs. academic vs. no concentration), personal characteristics embedded in a social cognitive theory framework (e.g., self-efficacy, academic expectations), and contextual variables (e.g., parental expectations, socioeconomic status [SES]) interact with each other in the prediction of students' income and job satisfaction 8 years after graduating from high school. Using a nationally representative data set (the Educational Longitudinal Study of 2002), we found significant differences by sex and course-taking pattern in the prediction of income: Among men, CTE concentrators had the highest income, whereas among women, academic concentrators reported the greatest earnings. We observed similar levels of job satisfaction among academic and CTE concentrators. We also found that SES significantly moderated the effect of English self-efficacy and academic expectations in the prediction of income and general effort in the prediction of job satisfaction. Our findings highlight how a social cognitive framework can be used to investigate the links between high school course-taking, personal and contextual factors, and job outcomes. They additionally suggest the need to consider a broader set of outcomes for evaluating the benefits of CTE participation.

Researchers, theorists, and practitioners have expressed a renewed interest in the longitudinal dynamics of personality characteristics in adulthood, including organic life span trajectories and their amenability to volitional change. However, this research has apparently not yet expanded to include the Dark Triad (psychopathy, narcissism, Machiavellianism), despite approximately 2 decades of research that has thoroughly examined other important issues related to construct validity and interpersonal behavior. We argue that researchers in postsecondary, occupational, and community-based settings are in a unique position to study the important phenomenon of Dark Triad malleability, as they are less hindered by obstacles in clinical and forensic contexts that have generated largely inconclusive results. In this article, we discuss several examples of methods for evaluating, quantifying, and interpreting Dark Triad malleability, examples of relevant extant training programs, possibilities for developing new programs, and factors that may moderate training efficacy, including Dark Triad levels themselves. Beyond addressing a fundamental question regarding the nature of these traits, the Dark Triad's destructive tendencies suggest that efforts to reduce them would provide myriad societal benefits and could propel Dark Triad research in an important new direction.

In this research report, we describe a study to map the scores of the TOEFL® Essentials™ test to the Canadian Language Benchmarks (CLB). The TOEFL Essentials test is a four-skills assessment of foundational English language skills and communication abilities in academic and general (daily life) contexts. At the time of writing this report, the test was the most recent addition to the TOEFL® Family of Assessments. TOEFL Essentials test scores are intended to provide academic programs and other users with reliable information regarding the test taker's ability to understand and use English. Mapping of scores to widely used language frameworks such as the CLB provides additional support for interpreting test results and for making inferences regarding test-taker abilities. The score mapping process consisted of the following steps, as recommended in the literature: (a) establishing construct congruence between the test content and the performance descriptors of the CLB; (b) establishing recommended minimum test scores (cut scores) required to classify language learners into CLB levels, based on the judgments of local experts; and (c) providing evidence of procedural, internal, and external validation of the recommended cut scores.