Journal of applied measurement最新文献

The estimates of intraclass correlations are known to be biased, but there are few analytical ways to assess the amount of bias. The analytical approach requires the normality assumption to estimate bias. Bootstrap requires no such assumption and can, therefore, be used to estimate bias, regardless of the model assumption. We utilize cluster bootstrapping to calculate the bias in estimating the intraclass correlation. A well-known dataset is provided to illustrate the bias estimation in a typical study design of intraclass correlation, and its implications for other study designs are also discussed.

The purpose of this study is to demonstrate an application of the many-facet Rasch model (MFRM) in evaluating the impromptu speech skills of pre-service principals in Taiwan. The findings showed that the topics of speech did not exhibit different difficulty measures. With respect to scoring criteria, time control was the most difficult aspect among the scoring criteria. Regarding gender difference in raters, female raters gave lower scores than male raters, but there was no statistical evidence for gender-related bias. However, raters exhibited statistically significant differences in rater severity. The results of this study demonstrates that the MFRM provides a scientific approach to assessment, which can reveal some useful diagnostic information from the original ordinal rating scores on impromptu speech.

Multidimensional pairwise comparison (MPC) items have been widely used to assess career interest, value and personality to avoid response bias in educational sectors. In reality, a statement in an MPC item may have different utilities for different groups, which is referred to as differential statement functioning (DSF). Few studies have been investigated DSF assessment. Based on a Rasch model for MPC items, this study adapts three methods to detect DSF for polytomous MPC items: the equal-mean-utility (EMU) method, the all-other-statement (AOS) method and the constant-statement (CS) method. Simulation study was conducted to evaluate the recovery of parameters as well as the performance of the proposed methods. Results showed that when the test contains DSF statement(s), the CS method where one or more DSF-free statements are chosen as an anchor will yield accurate estimates and perform well for DSF assessment. An empirical example of career interest assessment was provided. .

Streiner, Norman and Cairney (2015) "Health Measurement Scales: A practical guide to their development and use", now in its fifth edition, is one of the foundational texts of the health outcomes movement. It states that "the differences between scales constructed with IRT and CTT are trivial." (Streiner, Norman and Cairney, 2015, p. 299) This statement is representative of the view which emphasizes the equivalence of True-Score Theory (TST) (also known as Classical Test Theory [CTT]) and the Rasch Measurement Model [RMM]). This view is widely held and has been one factor in limiting the application of RMM in the development of health outcome measures. However, this equivalence view relies heavily on a paper by Fan (1998) which examined the item statistics derived from TST, IRT (Item Response Theory) and the RMM for a large educational dataset. While subject to a number of theoretical and practical criticisms from a RMM perspective this paper has not been replicated with a large sample. This paper by replicating and extending the paper by Fan (1998) challenges the finding that item difficulty indexes derived from high and low ability samples using TST techniques are invariant. They are not. On the other hand, item locations derived from the RMM have a high degree of invariance. This secondary data analysis, by working through the methods used by Fan (1998) also demonstrates that a reliance on the magnitude of correlational coefficients cannot be used to determine the invariance of item difficulty indexes. An investigation into the linearity of the correlations using scatter plots is also required. Finally, an item analysis derived from the item difficulty indexes which displays a picture of the test as a whole shows that, for this large sample, the differences between scales constructed with TST and the RMM are not trivial.

Many assessment scales in the social sciences are composed of multiple items that form a subscale structure. They have this structure because more than one aspect of the variable is assessed and more than one item assesses each aspect. Nevertheless, generally, a single measurement is required from the scale. A characteristic of this measurement is that the greater the number of items, and categories within an item, that assess an aspect, the greater its influence on the final measurement. One way to control this influence is to include the desired relative number of items and categories to assess each aspect in the scale. However, there are circumstances where designing the required number of items and categories for each aspect is challenging. This paper shows a method of controlling the influence of the number of items and categories assessing each aspect by a-priori weighting of items at the person measurement stage with the Rasch model.

The purpose of the present report was to assess congruence between a language-based national examination (termed English placement test - EPT) and the Common European Framework of Reference for Languages (CEFR) levels. To this end, a series of methodological steps were put forth to accumulate evidence suggesting that language performance based on the EPT instrument can be split onto meaningful subgroups based on theoretical (expert judgement on difficulty level and CEFR correspondence) and empirical considerations (i.e., how well these levels and subgroups emerged). Participants were 2642 high school graduates who took on the EPT instrument as part of their entry criteria to the university and for the purposes of the present study only the structure subscale is presented. Items were classified as reflecting specific CEFR levels and a person-based analysis attempted to classify individuals sharing the same behavioral patterns. Results using a latent class analysis (LCA) indicated that a Pre-A1, an A1 an A2 a B1 and a B2 levels were present with regard to the structure domain of language. Results showed a strong alignment between the EPT structure domain and CEFR guidelines using various methodological approaches.

A consensus has developed that high-quality teacher evaluation systems require multiple measures. We examine multiple measures from a large urban school district, which has included observational ratings and value-added ratings in its system since 2010. Evaluation systems that do not account for observer severity, classroom context, and other factors may yield different results from systems that do account for these factors. Choosing a simpler system involves a trade-off regarding a system's robustness or defensibility. Using a many-faceted Rasch model, we explore rating components like observer, time of year, and subdomain. We find high reliability of the resulting teacher ratings, some impact of adjusting for observer differences and differences between subdomains, and positive correlation with value-added measures. A comprehensive analysis like MFRM should be part of a district's evaluation system, even if only as a robustness check, and districts should examine how observational scores and classroom context are related.

Accurate parameter estimation in the Rasch model involves the assumption of conditional independence, also termed local independence. Conditional on ability, the responses to items A and B should be independent. Two types of conditional dependence are detailed in this pedagogical piece: trait dependency and response dependency. The bias in difficulty and reliability and the estimates of fit and correlated residuals resulting from these dependencies are compared and contrasted to results from using models that account for the dependency. Contrasts with results from a 2-parameter item response theory model are also briefly noted.

The emotional burden of living with Type 1 diabetes (T1D) is experienced differently in each life stage. Thus the measurement of diabetes distress (DD) warrants tailoring to particular developmental stages, specifically emerging adulthood (ages 18-30). The new measure entitled the Problem Areas in Diabetes- Emerging Adult version (PAID-EA) is intended to be a developmentally-embedded measure of DD for use in clinical and research settings. The goal of the present study was to use Rasch psychometric analysis to reduce and refine the PAID-EA. Emerging adults with T1D (n = 194) completed the 30-item online survey. Evaluation of response category functioning, measurement precision, redundancy, unidimensionality and targeting guided item reduction through iterative revisions. The reduced and refined PAID-EA consists of 25 items and shows promising utility for clinicians and researchers.