CD-CAT plays a significant role in diagnosing and assessing students, contributing significantly to improving teaching effectiveness. However, in classroom teaching scenarios, unlike large-scale assessments where a large number of samples can be used to accurately estimate item parameters, non-parametric CD-CAT becomes the only feasible choice. Compared to parametric CD-CAT, non-parametric CD-CAT started later, and research mainly focuses on non-parametric item selection strategies. However, the existing non-parametric item selection strategies have the disadvantage of low efficiency, and there is little research on non-parametric termination rules. Therefore, this study proposes two more efficient item selection strategies: Non-Parametric Dynamic Binary Search (NDBS) and General Non-Parametric Dynamic Binary Search (GNDBS), as well as a non-parametric termination rule：Non-parametric Dynamic Binary Searching Index (NDBI). Simulation results show: (1) Under all conditions, the pattern classification accuracy rate of NDBS is higher than that of NPS, so NDBS can be used as the item selection strategy when there are no samples available. (2) In most cases, the performance of GNDBS is better than other item selection strategies, so GNDBS can be chosen as the item selection strategy when there are few samples available. (3) In variable-length tests, when the research objective is to obtain more accurate classification results, the critical value of the NDBI rule can be reduced; conversely, the critical values of the NDBI and GNDBI rules can be appropriately increased.
{"title":"Non-Parametric CD-CAT Item Selection Strategy and Termination Rules Based on Binary Search Algorithm","authors":"Junjie Li, Huijing Zheng","doi":"10.59863/dkui7768","DOIUrl":"https://doi.org/10.59863/dkui7768","url":null,"abstract":"CD-CAT plays a significant role in diagnosing and assessing students, contributing significantly to improving teaching effectiveness. However, in classroom teaching scenarios, unlike large-scale assessments where a large number of samples can be used to accurately estimate item parameters, non-parametric CD-CAT becomes the only feasible choice. Compared to parametric CD-CAT, non-parametric CD-CAT started later, and research mainly focuses on non-parametric item selection strategies. However, the existing non-parametric item selection strategies have the disadvantage of low efficiency, and there is little research on non-parametric termination rules. Therefore, this study proposes two more efficient item selection strategies: Non-Parametric Dynamic Binary Search (NDBS) and General Non-Parametric Dynamic Binary Search (GNDBS), as well as a non-parametric termination rule：Non-parametric Dynamic Binary Searching Index (NDBI). Simulation results show: (1) Under all conditions, the pattern classification accuracy rate of NDBS is higher than that of NPS, so NDBS can be used as the item selection strategy when there are no samples available. (2) In most cases, the performance of GNDBS is better than other item selection strategies, so GNDBS can be chosen as the item selection strategy when there are few samples available. (3) In variable-length tests, when the research objective is to obtain more accurate classification results, the critical value of the NDBI rule can be reduced; conversely, the critical values of the NDBI and GNDBI rules can be appropriately increased.","PeriodicalId":72586,"journal":{"name":"Chinese/English journal of educational measurement and evaluation","volume":"269 1","pages":""},"PeriodicalIF":0.0,"publicationDate":"2024-04-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"140781312","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Ou Lydia Liu, Harrison Kell, Kevin Williams, Guangming Ling, Micah Sanders
In an era driven by rapid technological advancements human skills are becoming ever more important. As millions of workers reskill and upskill to meet the challenges of the modern workforce, it’s critical they understand what skills are being emphasized by employers and pathways leading to skills acquisition. This paper reviews influential frameworks for essential workforce skills and proposes the ETS Skills Taxonomy 2025. The Taxonomy brings a broad set of cognitive, interpersonal, intrapersonal, digital, and lifelong learning skills with definition and assessment considerations. In particular, it highlights new skills such as sciential skills, remote work, and coachability as a new workforce is being jointly shaped by shifting skills requirements and societal needs that call for inclusion and agility.
{"title":"ETS Skills Taxonomy","authors":"Ou Lydia Liu, Harrison Kell, Kevin Williams, Guangming Ling, Micah Sanders","doi":"10.59863/nmie9603","DOIUrl":"https://doi.org/10.59863/nmie9603","url":null,"abstract":"In an era driven by rapid technological advancements human skills are becoming ever more important. As millions of workers reskill and upskill to meet the challenges of the modern workforce, it’s critical they understand what skills are being emphasized by employers and pathways leading to skills acquisition. This paper reviews influential frameworks for essential workforce skills and proposes the ETS Skills Taxonomy 2025. The Taxonomy brings a broad set of cognitive, interpersonal, intrapersonal, digital, and lifelong learning skills with definition and assessment considerations. In particular, it highlights new skills such as sciential skills, remote work, and coachability as a new workforce is being jointly shaped by shifting skills requirements and societal needs that call for inclusion and agility.","PeriodicalId":72586,"journal":{"name":"Chinese/English journal of educational measurement and evaluation","volume":" 42","pages":""},"PeriodicalIF":0.0,"publicationDate":"2023-12-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"138619152","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
In educational evaluations at home and abroad, polytomous scoring items are becoming increasingly important. They can provide richer and more valuable information, with unmatched advantages compared to binary (0-1 scoring) items. If used as a tool for teachers to diagnose and assess students in the classroom, CD-CAT (Cognitive Diagnostic Computerized Adaptive Testing) has significant implications for improving teaching effectiveness. However, in classroom teaching situations, it is not feasible to estimate item parameters accurately with a large sample, as in large-scale assessments. In such cases, non-parametric CD-CAT becomes the only viable option. Compared to parametric CD-CAT, non-parametric CD-CAT started later and is particularly lacking in research related to polytomous scoring. Item selection method is at the core of CD-CAT, so it is essential to develop a non-parametric item selection method suitable for polytomous scoring CD-CAT. This study proposes a non-parametric item selection method for polytomous scoring cognitive diagnostic computerized adaptive testing (PCD-CAT): the Manhattan Distance Non-parametric Difference index item selection method (MD-NDI). The results of simulation studies indicate: (1) MD-NDI item selection method is suitable for PCD-CAT scenarios and exhibits better performance when the item bank quality is poor or the sample size for estimating item parameters is limited. (2) MD-NDI does not require pre-testing of items and distributes item usage more evenly, effectively ensuring the security of the item bank. (3) Even in cases of incorrectly specified item bank of Qc-matrix, MD-NDI still shows higher pattern correct classification rates. (4) In the study of variable-length PCD-CAT, MD-NDI not only reduces the test length in most conditions but also has a higher pattern correct classification rates when reaching the test termination rule.
{"title":"An Efficient Non-parametric Item Selection Method for Polytomous Scoring CD-CAT","authors":"Junjie Li, Jinghui Zheng, Chunhua Kang, Pingfei Zeng","doi":"10.59863/indp7038","DOIUrl":"https://doi.org/10.59863/indp7038","url":null,"abstract":"In educational evaluations at home and abroad, polytomous scoring items are becoming increasingly important. They can provide richer and more valuable information, with unmatched advantages compared to binary (0-1 scoring) items. If used as a tool for teachers to diagnose and assess students in the classroom, CD-CAT (Cognitive Diagnostic Computerized Adaptive Testing) has significant implications for improving teaching effectiveness. However, in classroom teaching situations, it is not feasible to estimate item parameters accurately with a large sample, as in large-scale assessments. In such cases, non-parametric CD-CAT becomes the only viable option. Compared to parametric CD-CAT, non-parametric CD-CAT started later and is particularly lacking in research related to polytomous scoring. Item selection method is at the core of CD-CAT, so it is essential to develop a non-parametric item selection method suitable for polytomous scoring CD-CAT. This study proposes a non-parametric item selection method for polytomous scoring cognitive diagnostic computerized adaptive testing (PCD-CAT): the Manhattan Distance Non-parametric Difference index item selection method (MD-NDI). The results of simulation studies indicate: (1) MD-NDI item selection method is suitable for PCD-CAT scenarios and exhibits better performance when the item bank quality is poor or the sample size for estimating item parameters is limited. (2) MD-NDI does not require pre-testing of items and distributes item usage more evenly, effectively ensuring the security of the item bank. (3) Even in cases of incorrectly specified item bank of Qc-matrix, MD-NDI still shows higher pattern correct classification rates. (4) In the study of variable-length PCD-CAT, MD-NDI not only reduces the test length in most conditions but also has a higher pattern correct classification rates when reaching the test termination rule.","PeriodicalId":72586,"journal":{"name":"Chinese/English journal of educational measurement and evaluation","volume":" 5","pages":""},"PeriodicalIF":0.0,"publicationDate":"2023-12-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"138615954","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
This mixed methods study compares how secondary school teachers implemented Education Reform in China in 2012 and 2022. The survey asked how often a teaching or evaluation strategy was used. The conclusions of the current study indicate that even though teacher-directed lessons (teacher talk, questioning, and discussions) still dominated, sample teachers were able to use a variety of student-centered learning (SCL) methods (activities and group work) in accordance with the Reform initiatives. A significant obstacle to reform remains high-stakes examinations, which rely heavily on rote memorization, rather than the creative application of knowledge.
{"title":"Revisiting Secondary Education Reform in China: Comparing the Perceptions of Teachers and Students between 2012 and 2022","authors":"Yixiang Jin, Yee Han Peter Joong, Rose Gibbs","doi":"10.59863/zuxn4915","DOIUrl":"https://doi.org/10.59863/zuxn4915","url":null,"abstract":"This mixed methods study compares how secondary school teachers implemented Education Reform in China in 2012 and 2022. The survey asked how often a teaching or evaluation strategy was used. The conclusions of the current study indicate that even though teacher-directed lessons (teacher talk, questioning, and discussions) still dominated, sample teachers were able to use a variety of student-centered learning (SCL) methods (activities and group work) in accordance with the Reform initiatives. A significant obstacle to reform remains high-stakes examinations, which rely heavily on rote memorization, rather than the creative application of knowledge.","PeriodicalId":72586,"journal":{"name":"Chinese/English journal of educational measurement and evaluation","volume":"30 1","pages":""},"PeriodicalIF":0.0,"publicationDate":"2023-12-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"138624046","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Chanjin Zheng, Shaoyang Guo, Wei Xia, Shaoguang Mao
For a long time, Chinese language teachers in primary and secondary schools have been confronting challenges of heavy workload, low efficiency, and difficulty in improving the quality of composition evaluations. This article introduces “ELion”, an intelligent Chinese composition tutoring system based on large language models. The system utilizes deep linguistic features to evaluate the quality of compositions and provide interpretable feedback. By discussing the overall design, evaluation framework structure, and scoring algorithm principles of ELion, this paper addresses the theoretical, technical, and engineering issues of intelligent evaluation of Chinese compositions in the educational context. Small-scale experiments conducted in schools demonstrate that ELion performs well in language error detection, rhetorical techniques, and the expression of actions and emotions. It can basically meet the needs of Chinese language teaching in primary and secondary schools. In the future, ELion will further develop algorithms for ”instruction-learning-evaluation” alignment assessment, and personalized precise feedback generation, based on the GPT model. This will improve the evaluation effectiveness in topic analysis, text structure, and genuine emotional expression. Additionally, systematic field experiments for the system will be conducted to explore the application of artificial intelligence in education.
{"title":"ELion: An Intelligent Chinese Composition Tutoring System Based on Large Language Models","authors":"Chanjin Zheng, Shaoyang Guo, Wei Xia, Shaoguang Mao","doi":"10.59863/mpjo6480","DOIUrl":"https://doi.org/10.59863/mpjo6480","url":null,"abstract":"For a long time, Chinese language teachers in primary and secondary schools have been confronting challenges of heavy workload, low efficiency, and difficulty in improving the quality of composition evaluations. This article introduces “ELion”, an intelligent Chinese composition tutoring system based on large language models. The system utilizes deep linguistic features to evaluate the quality of compositions and provide interpretable feedback. By discussing the overall design, evaluation framework structure, and scoring algorithm principles of ELion, this paper addresses the theoretical, technical, and engineering issues of intelligent evaluation of Chinese compositions in the educational context. Small-scale experiments conducted in schools demonstrate that ELion performs well in language error detection, rhetorical techniques, and the expression of actions and emotions. It can basically meet the needs of Chinese language teaching in primary and secondary schools. In the future, ELion will further develop algorithms for ”instruction-learning-evaluation” alignment assessment, and personalized precise feedback generation, based on the GPT model. This will improve the evaluation effectiveness in topic analysis, text structure, and genuine emotional expression. Additionally, systematic field experiments for the system will be conducted to explore the application of artificial intelligence in education.","PeriodicalId":72586,"journal":{"name":"Chinese/English journal of educational measurement and evaluation","volume":"51 1","pages":""},"PeriodicalIF":0.0,"publicationDate":"2023-09-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"77127152","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
In this paper, we present a novel method for detecting careless responses in a low-stakes practice exam using machine learning models. Rather than classifying test-taker responses as careless based on model fit statistics or knowledge of truth, we built a model to predict significant changes in test scores between a practice test and an official test based on attributes of practice test items. We extracted features from practice test items using hypotheses about how careless test takers respond to items and cross-validated model performance to optimize out-of-sample predictions and reduce heteroscedasticity when predicting the closest official test. All analyses use data from the practice and official versions of the Duolingo English Test. We discuss the implications of using a machine learning model for predicting careless cases as compared with alternative, popular methods.
{"title":"Detecting Careless Cases in Practice Tests","authors":"Steven Nydick","doi":"10.59863/lavm1367","DOIUrl":"https://doi.org/10.59863/lavm1367","url":null,"abstract":"In this paper, we present a novel method for detecting careless responses in a low-stakes practice exam using machine learning models. Rather than classifying test-taker responses as careless based on model fit statistics or knowledge of truth, we built a model to predict significant changes in test scores between a practice test and an official test based on attributes of practice test items. We extracted features from practice test items using hypotheses about how careless test takers respond to items and cross-validated model performance to optimize out-of-sample predictions and reduce heteroscedasticity when predicting the closest official test. All analyses use data from the practice and official versions of the Duolingo English Test. We discuss the implications of using a machine learning model for predicting careless cases as compared with alternative, popular methods.","PeriodicalId":72586,"journal":{"name":"Chinese/English journal of educational measurement and evaluation","volume":"60 1","pages":""},"PeriodicalIF":0.0,"publicationDate":"2023-09-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"139345244","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
京公网安备 11010802042870号
京ICP备2023020795号-1
扫码关注我们
求助内容：
应助结果提醒方式：