Job and practice analysis is a commonly used method for determining examination content specifications. However, difficulties arise when many domains are present, as mainstream approaches do not fully adhere to the essence of the weighing process, namely a “comparison‐evaluation‐decision” framework for assigning percentage values to the content. Stemming from the principle of comparing multiple criteria for making decisions, the Analytic Hierarchy Process (AHP) provides an appropriate solution that circumvents the aforementioned obstacle. We propose using an extended version of AHP called Group AHP (GAHP) to weight content specifications for standardized medical education assessment. Specifically, GAHP is integrated with the Delphi method and expected to aid exam developers in integrating feedback from diverse experienced physicians when determining content specifications for the National Medical Licensing Examination (NMLE) in China. The complete flow of the proposed approach was demonstrated in this study with an application to the NMLE.
{"title":"Weighting Content Specifications for the National Medical Licensing Examination via Group Analytic Hierarchy Process","authors":"Xiaomei Hong, Zhehan Jiang, Hanyu Liu, Fen Cai","doi":"10.1111/emip.12620","DOIUrl":"https://doi.org/10.1111/emip.12620","url":null,"abstract":"Job and practice analysis is a commonly used method for determining examination content specifications. However, difficulties arise when many domains are present, as mainstream approaches do not fully adhere to the essence of the weighing process, namely a “comparison‐evaluation‐decision” framework for assigning percentage values to the content. Stemming from the principle of comparing multiple criteria for making decisions, the Analytic Hierarchy Process (AHP) provides an appropriate solution that circumvents the aforementioned obstacle. We propose using an extended version of AHP called Group AHP (GAHP) to weight content specifications for standardized medical education assessment. Specifically, GAHP is integrated with the Delphi method and expected to aid exam developers in integrating feedback from diverse experienced physicians when determining content specifications for the National Medical Licensing Examination (NMLE) in China. The complete flow of the proposed approach was demonstrated in this study with an application to the NMLE.","PeriodicalId":516921,"journal":{"name":"Educational Measurement: Issues and Practice","volume":" April","pages":""},"PeriodicalIF":0.0,"publicationDate":"2024-07-19","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141823629","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Xuelan Qiu, Jimmy de la Torre, You‐Gan Wang, Jinran Wu
Multidimensional forced‐choice (MFC) items have been found to be useful to reduce response biases in personality assessments. However, conventional scoring methods for the MFC items result in ipsative data, hindering the wider applications of the MFC format. In the last decade, a number of item response theory (IRT) models have been developed, majority of which are for MFC items with binary responses. However, MFC items with polytomous responses are more informative and have many applications. This paper develops a polytomous Rasch ipsative model (pRIM) that can deal with ipsative data and yield estimates that measure construct differentiation—a latent trait that describes the degree to which the personality constructs (e.g., interests) distinguish between each other. The pRIM and its simpler form are applied to a career interests assessment containing four‐category MFC items and the measures of interests differentiation are used for both intra‐ and interpersonal comparisons. Simulations are conducted to examine the recovery of the parameters under various conditions. The results show that the parameters of the pRIM can be well recovered, particularly when a complete linking design and a large sample are used. The implications and application of the pRIM in the personality assessment using MFC items are discussed.
{"title":"Item Response Theory Models for Polytomous Multidimensional Forced‐Choice Items to Measure Construct Differentiation","authors":"Xuelan Qiu, Jimmy de la Torre, You‐Gan Wang, Jinran Wu","doi":"10.1111/emip.12621","DOIUrl":"https://doi.org/10.1111/emip.12621","url":null,"abstract":"Multidimensional forced‐choice (MFC) items have been found to be useful to reduce response biases in personality assessments. However, conventional scoring methods for the MFC items result in ipsative data, hindering the wider applications of the MFC format. In the last decade, a number of item response theory (IRT) models have been developed, majority of which are for MFC items with binary responses. However, MFC items with polytomous responses are more informative and have many applications. This paper develops a polytomous Rasch ipsative model (pRIM) that can deal with ipsative data and yield estimates that measure construct differentiation—a latent trait that describes the degree to which the personality constructs (e.g., interests) distinguish between each other. The pRIM and its simpler form are applied to a career interests assessment containing four‐category MFC items and the measures of interests differentiation are used for both intra‐ and interpersonal comparisons. Simulations are conducted to examine the recovery of the parameters under various conditions. The results show that the parameters of the pRIM can be well recovered, particularly when a complete linking design and a large sample are used. The implications and application of the pRIM in the personality assessment using MFC items are discussed.","PeriodicalId":516921,"journal":{"name":"Educational Measurement: Issues and Practice","volume":"111 45","pages":""},"PeriodicalIF":0.0,"publicationDate":"2024-06-10","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141362892","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
京公网安备 11010802042870号
京ICP备2023020795号-1
扫码关注我们
求助内容：
应助结果提醒方式：