首页 > 最新文献

Journal of Educational and Behavioral Statistics最新文献

英文 中文
Erratum to Identifying Informative Predictor Variables With Random Forests 对《用随机森林识别信息预测变量》的勘误
IF 2.4 3区 心理学 Q2 EDUCATION & EDUCATIONAL RESEARCH Pub Date : 2023-10-13 DOI: 10.3102/10769986231204871
{"title":"Erratum to Identifying Informative Predictor Variables With Random Forests","authors":"","doi":"10.3102/10769986231204871","DOIUrl":"https://doi.org/10.3102/10769986231204871","url":null,"abstract":"","PeriodicalId":48001,"journal":{"name":"Journal of Educational and Behavioral Statistics","volume":"33 1","pages":""},"PeriodicalIF":2.4,"publicationDate":"2023-10-13","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"139319822","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 0
A Multistrategy Cognitive Diagnosis Model Incorporating Item Response Times Based on Strategy Selection Theories 基于策略选择理论的包含项目反应时间的多策略认知诊断模型
3区 心理学 Q2 EDUCATION & EDUCATIONAL RESEARCH Pub Date : 2023-10-02 DOI: 10.3102/10769986231200469
Junhuan Wei, Liufen Luo, Yan Cai, Dongbo Tu
Response times (RTs) facilitate the quantification of underlying cognitive processes in problem-solving behavior. To provide more comprehensive diagnostic feedback on strategy selection and attribute profiles with multistrategy cognitive diagnosis model (CDM) and utilize additional information for item RTs, this study develops a multistrategy cognitive diagnosis modeling framework combined with RTs. The proposed model integrates individual response accuracy and RT into a unified framework to define strategy selection and make it closer to the individual’s strategy selection process. Simulation studies demonstrated that the proposed model had reasonable parameter recovery and attribute classification accuracy and outperformed the existing multistrategy CDMs and single-strategy CDMs in terms of performance. Empirical results further illustrated the practical application and the advantages of the proposed model.
反应时间(RTs)促进了问题解决行为中潜在认知过程的量化。为了利用多策略认知诊断模型(CDM)对策略选择和属性概况提供更全面的诊断反馈,并利用项目RTs的附加信息,本研究开发了一个结合RTs的多策略认知诊断建模框架。该模型将个体反应准确性和RT整合到一个统一的框架中来定义策略选择,使其更接近个体的策略选择过程。仿真研究表明,该模型具有合理的参数恢复和属性分类精度,性能优于现有的多策略cdm和单策略cdm。实证结果进一步说明了该模型的实际应用和优势。
{"title":"A Multistrategy Cognitive Diagnosis Model Incorporating Item Response Times Based on Strategy Selection Theories","authors":"Junhuan Wei, Liufen Luo, Yan Cai, Dongbo Tu","doi":"10.3102/10769986231200469","DOIUrl":"https://doi.org/10.3102/10769986231200469","url":null,"abstract":"Response times (RTs) facilitate the quantification of underlying cognitive processes in problem-solving behavior. To provide more comprehensive diagnostic feedback on strategy selection and attribute profiles with multistrategy cognitive diagnosis model (CDM) and utilize additional information for item RTs, this study develops a multistrategy cognitive diagnosis modeling framework combined with RTs. The proposed model integrates individual response accuracy and RT into a unified framework to define strategy selection and make it closer to the individual’s strategy selection process. Simulation studies demonstrated that the proposed model had reasonable parameter recovery and attribute classification accuracy and outperformed the existing multistrategy CDMs and single-strategy CDMs in terms of performance. Empirical results further illustrated the practical application and the advantages of the proposed model.","PeriodicalId":48001,"journal":{"name":"Journal of Educational and Behavioral Statistics","volume":"27 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-10-02","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"135899998","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 0
Utilizing Real-Time Test Data to Solve Attenuation Paradox in Computerized Adaptive Testing to Enhance Optimal Design 利用实时测试数据解决计算机自适应测试中的衰减悖论,提高优化设计
3区 心理学 Q2 EDUCATION & EDUCATIONAL RESEARCH Pub Date : 2023-09-25 DOI: 10.3102/10769986231197666
Jyun-Hong Chen, Hsiu-Yi Chao
To solve the attenuation paradox in computerized adaptive testing (CAT), this study proposes an item selection method, the integer programming approach based on real-time test data (IPRD), to improve test efficiency. The IPRD method turns information regarding the ability distribution of the population from real-time test data into feasible test constraints to reversely assembled shadow tests for item selection to prevent the attenuation paradox by integer programming. A simulation study was conducted to thoroughly investigate IPRD performance. The results indicate that the IPRD method can efficiently improve CAT performance in terms of the precision of trait estimation and satisfaction of all required test constraints, especially for conditions with stringent exposure control.
为了解决计算机自适应测试(CAT)中的衰减悖论,提出了一种基于实时测试数据的整数规划法(IPRD)项目选择方法,以提高测试效率。IPRD方法通过整数规划,将实时测试数据中关于总体能力分布的信息转化为可行的测试约束,反向组合阴影测试进行项目选择,防止衰减悖论。我们进行了一项模拟研究,以彻底调查知识产权开发的性能。结果表明,IPRD方法在特征估计精度和满足所有要求的测试约束方面可以有效地提高CAT性能,特别是在严格暴露控制的条件下。
{"title":"Utilizing Real-Time Test Data to Solve Attenuation Paradox in Computerized Adaptive Testing to Enhance Optimal Design","authors":"Jyun-Hong Chen, Hsiu-Yi Chao","doi":"10.3102/10769986231197666","DOIUrl":"https://doi.org/10.3102/10769986231197666","url":null,"abstract":"To solve the attenuation paradox in computerized adaptive testing (CAT), this study proposes an item selection method, the integer programming approach based on real-time test data (IPRD), to improve test efficiency. The IPRD method turns information regarding the ability distribution of the population from real-time test data into feasible test constraints to reversely assembled shadow tests for item selection to prevent the attenuation paradox by integer programming. A simulation study was conducted to thoroughly investigate IPRD performance. The results indicate that the IPRD method can efficiently improve CAT performance in terms of the precision of trait estimation and satisfaction of all required test constraints, especially for conditions with stringent exposure control.","PeriodicalId":48001,"journal":{"name":"Journal of Educational and Behavioral Statistics","volume":"12 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-09-25","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"135864451","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 0
Identifying Informative Predictor Variables With Random Forests 用随机森林识别信息预测变量
IF 2.4 3区 心理学 Q2 EDUCATION & EDUCATIONAL RESEARCH Pub Date : 2023-09-05 DOI: 10.3102/10769986231193327
Yannick Rothacher, Carolin Strobl
Random forests are a nonparametric machine learning method, which is currently gaining popularity in the behavioral sciences. Despite random forests’ potential advantages over more conventional statistical methods, a remaining question is how reliably informative predictor variables can be identified by means of random forests. The present study aims at giving a comprehensible introduction to the topic of variable selection with random forests and providing an overview of the currently proposed selection methods. Using simulation studies, the variable selection methods are examined regarding their statistical properties, and comparisons between their performances and the performance of a conventional linear model are drawn. Advantages and disadvantages of the examined methods are discussed, and practical recommendations for the use of random forests for variable selection are given.
随机森林是一种非参数机器学习方法,目前在行为科学中越来越受欢迎。尽管随机森林与更传统的统计方法相比具有潜在的优势,但剩下的问题是如何通过随机森林来确定可靠的信息预测变量。本研究旨在对随机森林的变量选择主题进行全面介绍,并概述目前提出的选择方法。通过仿真研究,对变量选择方法的统计特性进行了检验,并将其性能与传统线性模型的性能进行了比较。讨论了所检查方法的优缺点,并给出了使用随机森林进行变量选择的实用建议。
{"title":"Identifying Informative Predictor Variables With Random Forests","authors":"Yannick Rothacher, Carolin Strobl","doi":"10.3102/10769986231193327","DOIUrl":"https://doi.org/10.3102/10769986231193327","url":null,"abstract":"Random forests are a nonparametric machine learning method, which is currently gaining popularity in the behavioral sciences. Despite random forests’ potential advantages over more conventional statistical methods, a remaining question is how reliably informative predictor variables can be identified by means of random forests. The present study aims at giving a comprehensible introduction to the topic of variable selection with random forests and providing an overview of the currently proposed selection methods. Using simulation studies, the variable selection methods are examined regarding their statistical properties, and comparisons between their performances and the performance of a conventional linear model are drawn. Advantages and disadvantages of the examined methods are discussed, and practical recommendations for the use of random forests for variable selection are given.","PeriodicalId":48001,"journal":{"name":"Journal of Educational and Behavioral Statistics","volume":" ","pages":""},"PeriodicalIF":2.4,"publicationDate":"2023-09-05","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"45769538","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 0
Evaluating Psychometric Differences Between Fast Versus Slow Responses on Rating Scale Items 评估快速反应与慢速反应在评定量表项目上的心理测量差异
IF 2.4 3区 心理学 Q2 EDUCATION & EDUCATIONAL RESEARCH Pub Date : 2023-08-30 DOI: 10.3102/10769986231195260
N. Kim, D. Bolt
Some previous studies suggest that response times (RTs) on rating scale items can be informative about the content trait, but a more recent study suggests they may also be reflective of response styles. The latter result raises questions about the possible consideration of RTs for content trait estimation, as response styles are generally viewed as nuisance dimensions in the measurement of noncognitive constructs. In this article, we extend previous work exploring the simultaneous relevance of content and response style traits on RTs in self-report rating scale measurement by examining psychometric differences related to fast versus slow item responses. Following a parallel methodology applied with cognitive measures, we provide empirical illustrations of how RTs appear to be simultaneously reflective of both content and response style traits. Our results demonstrate that respondents may exhibit different response behaviors for fast versus slow responses and that both the content trait and response styles are relevant to such heterogeneity. These findings suggest that using RTs as a basis for improving the estimation of noncognitive constructs likely requires simultaneously attending to the effects of response styles.
先前的一些研究表明,评分量表项目的反应时间(RT)可以提供有关内容特征的信息,但最近的一项研究表明,它们也可能反映反应风格。后一个结果提出了关于在内容特征估计中可能考虑RT的问题,因为在非认知结构的测量中,反应风格通常被视为令人讨厌的维度。在这篇文章中,我们扩展了先前的工作,通过检查与快速和慢速项目反应相关的心理测量差异,探索内容和反应风格特征在自我报告评定量表测量中对RT的同时相关性。根据与认知测量相结合的平行方法,我们提供了RT如何同时反映内容和反应风格特征的实证说明。我们的研究结果表明,受访者对快速和慢速反应可能表现出不同的反应行为,内容特征和反应风格都与这种异质性有关。这些发现表明,使用RT作为改进非认知结构估计的基础,可能需要同时注意反应风格的影响。
{"title":"Evaluating Psychometric Differences Between Fast Versus Slow Responses on Rating Scale Items","authors":"N. Kim, D. Bolt","doi":"10.3102/10769986231195260","DOIUrl":"https://doi.org/10.3102/10769986231195260","url":null,"abstract":"Some previous studies suggest that response times (RTs) on rating scale items can be informative about the content trait, but a more recent study suggests they may also be reflective of response styles. The latter result raises questions about the possible consideration of RTs for content trait estimation, as response styles are generally viewed as nuisance dimensions in the measurement of noncognitive constructs. In this article, we extend previous work exploring the simultaneous relevance of content and response style traits on RTs in self-report rating scale measurement by examining psychometric differences related to fast versus slow item responses. Following a parallel methodology applied with cognitive measures, we provide empirical illustrations of how RTs appear to be simultaneously reflective of both content and response style traits. Our results demonstrate that respondents may exhibit different response behaviors for fast versus slow responses and that both the content trait and response styles are relevant to such heterogeneity. These findings suggest that using RTs as a basis for improving the estimation of noncognitive constructs likely requires simultaneously attending to the effects of response styles.","PeriodicalId":48001,"journal":{"name":"Journal of Educational and Behavioral Statistics","volume":" ","pages":""},"PeriodicalIF":2.4,"publicationDate":"2023-08-30","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"46284745","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 0
Cross-Classified Item Response Theory Modeling With an Application to Student Evaluation of Teaching 交叉分类项目反应理论模型及其在学生教学评价中的应用
IF 2.4 3区 心理学 Q2 EDUCATION & EDUCATIONAL RESEARCH Pub Date : 2023-08-24 DOI: 10.3102/10769986231193351
Sijia Huang, Li Cai
The cross-classified data structure is ubiquitous in education, psychology, and health outcome sciences. In these areas, assessment instruments that are made up of multiple items are frequently used to measure latent constructs. The presence of both the cross-classified structure and multivariate categorical outcomes leads to the so-called item-level data with cross-classified structure. An example of such data structure is the routinely collected student evaluation of teaching (SET) data. Motivated by the lack of research on multilevel IRT modeling with crossed random effects and the need of an approach that can properly handle SET data, this study proposed a cross-classified IRT model, which takes into account both the cross-classified data structure and properties of multiple items in an assessment instrument. A new variant of the Metropolis–Hastings Robbins–Monro (MH-RM) algorithm was introduced to address the computational complexities in estimating the proposed model. A preliminary simulation study was conducted to evaluate the performance of the algorithm for fitting the proposed model to data. The results indicated that model parameters were well recovered. The proposed model was also applied to SET data collected at a large public university to answer empirical research questions. Limitations and future research directions were discussed.
交叉分类的数据结构在教育、心理学和健康结果科学中无处不在。在这些领域,经常使用由多个项目组成的评估工具来衡量潜在的结构。交叉分类结构和多变量分类结果的存在导致了具有交叉分类结构的所谓项目级数据。这种数据结构的一个例子是常规收集的学生教学评估(SET)数据。由于缺乏对具有交叉随机效应的多级IRT建模的研究,并且需要一种能够正确处理SET数据的方法,本研究提出了一种交叉分类的IRT模型,该模型考虑了交叉分类的数据结构和评估工具中多个项目的特性。引入了Metropolis–Hastings-Robbins–Monro(MH-RM)算法的新变体,以解决估计所提出模型时的计算复杂性。进行了初步的模拟研究,以评估算法的性能,使所提出的模型与数据拟合。结果表明,模型参数恢复良好。所提出的模型也应用于在一所大型公立大学收集的SET数据,以回答实证研究问题。讨论了局限性和未来的研究方向。
{"title":"Cross-Classified Item Response Theory Modeling With an Application to Student Evaluation of Teaching","authors":"Sijia Huang, Li Cai","doi":"10.3102/10769986231193351","DOIUrl":"https://doi.org/10.3102/10769986231193351","url":null,"abstract":"The cross-classified data structure is ubiquitous in education, psychology, and health outcome sciences. In these areas, assessment instruments that are made up of multiple items are frequently used to measure latent constructs. The presence of both the cross-classified structure and multivariate categorical outcomes leads to the so-called item-level data with cross-classified structure. An example of such data structure is the routinely collected student evaluation of teaching (SET) data. Motivated by the lack of research on multilevel IRT modeling with crossed random effects and the need of an approach that can properly handle SET data, this study proposed a cross-classified IRT model, which takes into account both the cross-classified data structure and properties of multiple items in an assessment instrument. A new variant of the Metropolis–Hastings Robbins–Monro (MH-RM) algorithm was introduced to address the computational complexities in estimating the proposed model. A preliminary simulation study was conducted to evaluate the performance of the algorithm for fitting the proposed model to data. The results indicated that model parameters were well recovered. The proposed model was also applied to SET data collected at a large public university to answer empirical research questions. Limitations and future research directions were discussed.","PeriodicalId":48001,"journal":{"name":"Journal of Educational and Behavioral Statistics","volume":" ","pages":""},"PeriodicalIF":2.4,"publicationDate":"2023-08-24","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"45352027","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 0
IRT Models for Learning With Item-Specific Learning Parameters 具有项目特定学习参数的IRT学习模型
IF 2.4 3区 心理学 Q2 EDUCATION & EDUCATIONAL RESEARCH Pub Date : 2023-08-20 DOI: 10.3102/10769986231193096
Albert Yu, J. Douglas
We propose a new item response theory growth model with item-specific learning parameters, or ISLP, and two variations of this model. In the ISLP model, either items or blocks of items have their own learning parameters. This model may be used to improve the efficiency of learning in a formative assessment. We show ways that the ISLP model’s learning parameters can be estimated in simulation using Markov chain Monte Carlo (MCMC), demonstrate a way that the model could be used in the context of adaptive item selection to increase the rate of learning, and estimate the learning parameters in an empirical data analysis using the ISLP. In the simulation studies, the one-parameter logistic model was used as the measurement model to generate random response data with various test lengths and sample sizes. Ability growth was modeled with a few variations of the ISLP model, and it was verified that the parameters were accurately recovered. Secondly, we generated data using the linear logistic test model with known Q-matrix structure for the item difficulties. Using a two-step procedure gave very comparable results for the estimation of the learning parameters even when item difficulties were unknown. The potential benefit of using an adaptive selection method in conjunction with the ISLP model was shown by comparing total improvement in the examinees’ ability parameter to two other methods of item selection that do not utilize this growth model. If the ISLP holds, adaptive item selection consistently led to larger improvements over the other methods. A real data application of the ISLP was given to illustrate its use in a spatial reasoning study designed to promote learning. In this study, interventions were given after each block of ten items to increase ability. Learning parameters were estimated using MCMC.
我们提出了一种新的具有项目特定学习参数的项目反应理论增长模型,简称ISLP,以及该模型的两个变体。在ISLP模型中,项目或项目块都有自己的学习参数。该模型可用于提高形成性评估中的学习效率。我们展示了使用马尔可夫链蒙特卡罗(MCMC)在模拟中估计ISLP模型学习参数的方法,展示了该模型可用于自适应项目选择以提高学习率的方法,并在使用ISLP的经验数据分析中估计学习参数。在模拟研究中,使用单参数逻辑模型作为测量模型来生成具有不同测试长度和样本量的随机响应数据。用ISLP模型的一些变体对能力增长进行了建模,并验证了参数的准确恢复。其次,我们使用已知Q矩阵结构的线性逻辑测试模型生成项目困难的数据。即使在项目难度未知的情况下,使用两步程序对学习参数的估计也给出了非常相似的结果。通过将考生能力参数的总体改善与其他两种不使用该增长模型的项目选择方法进行比较,显示了将自适应选择方法与ISLP模型结合使用的潜在好处。如果ISLP成立,则自适应项目选择始终比其他方法有更大的改进。给出了ISLP的实际数据应用,以说明其在旨在促进学习的空间推理研究中的应用。在这项研究中,在每10个项目的区块后进行干预,以提高能力。使用MCMC估计学习参数。
{"title":"IRT Models for Learning With Item-Specific Learning Parameters","authors":"Albert Yu, J. Douglas","doi":"10.3102/10769986231193096","DOIUrl":"https://doi.org/10.3102/10769986231193096","url":null,"abstract":"We propose a new item response theory growth model with item-specific learning parameters, or ISLP, and two variations of this model. In the ISLP model, either items or blocks of items have their own learning parameters. This model may be used to improve the efficiency of learning in a formative assessment. We show ways that the ISLP model’s learning parameters can be estimated in simulation using Markov chain Monte Carlo (MCMC), demonstrate a way that the model could be used in the context of adaptive item selection to increase the rate of learning, and estimate the learning parameters in an empirical data analysis using the ISLP. In the simulation studies, the one-parameter logistic model was used as the measurement model to generate random response data with various test lengths and sample sizes. Ability growth was modeled with a few variations of the ISLP model, and it was verified that the parameters were accurately recovered. Secondly, we generated data using the linear logistic test model with known Q-matrix structure for the item difficulties. Using a two-step procedure gave very comparable results for the estimation of the learning parameters even when item difficulties were unknown. The potential benefit of using an adaptive selection method in conjunction with the ISLP model was shown by comparing total improvement in the examinees’ ability parameter to two other methods of item selection that do not utilize this growth model. If the ISLP holds, adaptive item selection consistently led to larger improvements over the other methods. A real data application of the ISLP was given to illustrate its use in a spatial reasoning study designed to promote learning. In this study, interventions were given after each block of ten items to increase ability. Learning parameters were estimated using MCMC.","PeriodicalId":48001,"journal":{"name":"Journal of Educational and Behavioral Statistics","volume":" ","pages":""},"PeriodicalIF":2.4,"publicationDate":"2023-08-20","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"44586129","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 1
DINA-BAG: A Bagging Algorithm for DINA Model Parameter Estimation in Small Samples 小样本条件下DINA模型参数估计的Bagging算法
IF 2.4 3区 心理学 Q2 EDUCATION & EDUCATIONAL RESEARCH Pub Date : 2023-08-07 DOI: 10.3102/10769986231188442
D. Arthur, Hua-Hua Chang
Cognitive diagnosis models (CDMs) are the assessment tools that provide valuable formative feedback about skill mastery at both the individual and population level. Recent work has explored the performance of CDMs with small sample sizes but has focused solely on the estimates of individual profiles. The current research focuses on obtaining accurate estimates of skill mastery at the population level. We introduce a novel algorithm (bagging algorithm for deterministic inputs noisy “and” gate) that is inspired by ensemble learning methods in the machine learning literature and produces more stable and accurate estimates of the population skill mastery profile distribution for small sample sizes. Using both simulated data and real data from the Examination for the Certificate of Proficiency in English, we demonstrate that the proposed method outperforms other methods on several metrics in a wide variety of scenarios.
认知诊断模型(CDMs)是一种评估工具,可以在个人和群体水平上提供关于技能掌握的有价值的形成性反馈。最近的工作已经在小样本量的情况下探索了cdm的性能,但是只关注于个体概况的估计。目前的研究重点是在人口水平上获得对技能掌握程度的准确估计。我们引入了一种新的算法(用于确定性输入噪声和门的bagging算法),该算法受到机器学习文献中的集成学习方法的启发,并对小样本量的总体技能掌握概况分布产生更稳定和准确的估计。使用英语水平证书考试的模拟数据和真实数据,我们证明了所提出的方法在各种场景下的几个指标上优于其他方法。
{"title":"DINA-BAG: A Bagging Algorithm for DINA Model Parameter Estimation in Small Samples","authors":"D. Arthur, Hua-Hua Chang","doi":"10.3102/10769986231188442","DOIUrl":"https://doi.org/10.3102/10769986231188442","url":null,"abstract":"Cognitive diagnosis models (CDMs) are the assessment tools that provide valuable formative feedback about skill mastery at both the individual and population level. Recent work has explored the performance of CDMs with small sample sizes but has focused solely on the estimates of individual profiles. The current research focuses on obtaining accurate estimates of skill mastery at the population level. We introduce a novel algorithm (bagging algorithm for deterministic inputs noisy “and” gate) that is inspired by ensemble learning methods in the machine learning literature and produces more stable and accurate estimates of the population skill mastery profile distribution for small sample sizes. Using both simulated data and real data from the Examination for the Certificate of Proficiency in English, we demonstrate that the proposed method outperforms other methods on several metrics in a wide variety of scenarios.","PeriodicalId":48001,"journal":{"name":"Journal of Educational and Behavioral Statistics","volume":" ","pages":""},"PeriodicalIF":2.4,"publicationDate":"2023-08-07","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"47475754","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 0
Bayesian Change-Point Analysis Approach to Detecting Aberrant Test-Taking Behavior Using Response Times 利用响应时间检测异常考生行为的贝叶斯变点分析方法
IF 2.4 3区 心理学 Q2 EDUCATION & EDUCATIONAL RESEARCH Pub Date : 2023-07-24 DOI: 10.3102/10769986231151961
Hongyue Zhu, Hong Jiao, Wei Gao, Xiangbin Meng
Change-point analysis (CPA) is a method for detecting abrupt changes in parameter(s) underlying a sequence of random variables. It has been applied to detect examinees’ aberrant test-taking behavior by identifying abrupt test performance change. Previous studies utilized maximum likelihood estimations of ability parameters, focusing on detecting one change point for each examinee. This article proposes a Bayesian CPA procedure using response times (RTs) to detect abrupt changes in examinee speed, which may be related to aberrant responding behaviors. The lognormal RT model is used to derive a procedure for detecting aberrant RT patterns. The method takes the numbers and locations of the change points as parameters in the model to detect multiple change points or multiple aberrant behaviors. Given the change points, the corresponding speed of each segment in the test can be estimated, which enables more accurate inferences about aberrant behaviors. Simulation study results indicate that the proposed procedure can effectively detect simulated aberrant behaviors and estimate change points accurately. The method is applied to data from a high-stakes computerized adaptive test, where its applicability is demonstrated.
变点分析(CPA)是一种用于检测随机变量序列下的参数突变的方法。它已被应用于通过识别考试成绩的突然变化来检测考生的异常考试行为。先前的研究使用了能力参数的最大似然估计,重点是检测每个受试者的一个变化点。本文提出了一种贝叶斯CPA程序,使用响应时间(RT)来检测考生速度的突然变化,这可能与异常反应行为有关。对数正态RT模型用于推导用于检测异常RT模式的程序。该方法以变化点的数量和位置作为模型中的参数来检测多个变化点或多个异常行为。给定变化点,可以估计测试中每个片段的相应速度,从而能够更准确地推断异常行为。仿真研究结果表明,该方法能够有效地检测模拟的异常行为,准确地估计变化点。该方法应用于高风险计算机自适应测试的数据,并证明了其适用性。
{"title":"Bayesian Change-Point Analysis Approach to Detecting Aberrant Test-Taking Behavior Using Response Times","authors":"Hongyue Zhu, Hong Jiao, Wei Gao, Xiangbin Meng","doi":"10.3102/10769986231151961","DOIUrl":"https://doi.org/10.3102/10769986231151961","url":null,"abstract":"Change-point analysis (CPA) is a method for detecting abrupt changes in parameter(s) underlying a sequence of random variables. It has been applied to detect examinees’ aberrant test-taking behavior by identifying abrupt test performance change. Previous studies utilized maximum likelihood estimations of ability parameters, focusing on detecting one change point for each examinee. This article proposes a Bayesian CPA procedure using response times (RTs) to detect abrupt changes in examinee speed, which may be related to aberrant responding behaviors. The lognormal RT model is used to derive a procedure for detecting aberrant RT patterns. The method takes the numbers and locations of the change points as parameters in the model to detect multiple change points or multiple aberrant behaviors. Given the change points, the corresponding speed of each segment in the test can be estimated, which enables more accurate inferences about aberrant behaviors. Simulation study results indicate that the proposed procedure can effectively detect simulated aberrant behaviors and estimate change points accurately. The method is applied to data from a high-stakes computerized adaptive test, where its applicability is demonstrated.","PeriodicalId":48001,"journal":{"name":"Journal of Educational and Behavioral Statistics","volume":"48 1","pages":"490 - 520"},"PeriodicalIF":2.4,"publicationDate":"2023-07-24","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"46477500","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 0
An Improved Inferential Procedure to Evaluate Item Discriminations in a Conditional Maximum Likelihood Framework 条件最大似然框架中评价项目判别的一种改进推理方法
IF 2.4 3区 心理学 Q2 EDUCATION & EDUCATIONAL RESEARCH Pub Date : 2023-07-19 DOI: 10.3102/10769986231183335
Clemens Draxler, A. Kurz, Can Gürer, J. Nolte
A modified and improved inductive inferential approach to evaluate item discriminations in a conditional maximum likelihood and Rasch modeling framework is suggested. The new approach involves the derivation of four hypothesis tests. It implies a linear restriction of the assumed set of probability distributions in the classical approach that represents scenarios of different item discriminations in a straightforward and efficient manner. Its improvement is discussed, compared to classical procedures (tests and information criteria), and illustrated in Monte Carlo experiments as well as real data examples from educational research. The results show an improvement of power of the modified tests of up to 0.3.
提出了一种改进的归纳推理方法,用于在条件最大似然和Rasch建模框架下评估项目判别。新方法涉及四个假设检验的推导。它意味着对经典方法中假设的概率分布集的线性限制,该方法以直接有效的方式表示不同项目判别的场景。与经典程序(测试和信息标准)相比,讨论了它的改进,并在蒙特卡洛实验和教育研究的真实数据示例中进行了说明。结果表明,改进试验的功率提高了0.3。
{"title":"An Improved Inferential Procedure to Evaluate Item Discriminations in a Conditional Maximum Likelihood Framework","authors":"Clemens Draxler, A. Kurz, Can Gürer, J. Nolte","doi":"10.3102/10769986231183335","DOIUrl":"https://doi.org/10.3102/10769986231183335","url":null,"abstract":"A modified and improved inductive inferential approach to evaluate item discriminations in a conditional maximum likelihood and Rasch modeling framework is suggested. The new approach involves the derivation of four hypothesis tests. It implies a linear restriction of the assumed set of probability distributions in the classical approach that represents scenarios of different item discriminations in a straightforward and efficient manner. Its improvement is discussed, compared to classical procedures (tests and information criteria), and illustrated in Monte Carlo experiments as well as real data examples from educational research. The results show an improvement of power of the modified tests of up to 0.3.","PeriodicalId":48001,"journal":{"name":"Journal of Educational and Behavioral Statistics","volume":" ","pages":""},"PeriodicalIF":2.4,"publicationDate":"2023-07-19","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"42084822","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 0
期刊
Journal of Educational and Behavioral Statistics
全部 Acc. Chem. Res. ACS Applied Bio Materials ACS Appl. Electron. Mater. ACS Appl. Energy Mater. ACS Appl. Mater. Interfaces ACS Appl. Nano Mater. ACS Appl. Polym. Mater. ACS BIOMATER-SCI ENG ACS Catal. ACS Cent. Sci. ACS Chem. Biol. ACS Chemical Health & Safety ACS Chem. Neurosci. ACS Comb. Sci. ACS Earth Space Chem. ACS Energy Lett. ACS Infect. Dis. ACS Macro Lett. ACS Mater. Lett. ACS Med. Chem. Lett. ACS Nano ACS Omega ACS Photonics ACS Sens. ACS Sustainable Chem. Eng. ACS Synth. Biol. Anal. Chem. BIOCHEMISTRY-US Bioconjugate Chem. BIOMACROMOLECULES Chem. Res. Toxicol. Chem. Rev. Chem. Mater. CRYST GROWTH DES ENERG FUEL Environ. Sci. Technol. Environ. Sci. Technol. Lett. Eur. J. Inorg. Chem. IND ENG CHEM RES Inorg. Chem. J. Agric. Food. Chem. J. Chem. Eng. Data J. Chem. Educ. J. Chem. Inf. Model. J. Chem. Theory Comput. J. Med. Chem. J. Nat. Prod. J PROTEOME RES J. Am. Chem. Soc. LANGMUIR MACROMOLECULES Mol. Pharmaceutics Nano Lett. Org. Lett. ORG PROCESS RES DEV ORGANOMETALLICS J. Org. Chem. J. Phys. Chem. J. Phys. Chem. A J. Phys. Chem. B J. Phys. Chem. C J. Phys. Chem. Lett. Analyst Anal. Methods Biomater. Sci. Catal. Sci. Technol. Chem. Commun. Chem. Soc. Rev. CHEM EDUC RES PRACT CRYSTENGCOMM Dalton Trans. Energy Environ. Sci. ENVIRON SCI-NANO ENVIRON SCI-PROC IMP ENVIRON SCI-WAT RES Faraday Discuss. Food Funct. Green Chem. Inorg. Chem. Front. Integr. Biol. J. Anal. At. Spectrom. J. Mater. Chem. A J. Mater. Chem. B J. Mater. Chem. C Lab Chip Mater. Chem. Front. Mater. Horiz. MEDCHEMCOMM Metallomics Mol. Biosyst. Mol. Syst. Des. Eng. Nanoscale Nanoscale Horiz. Nat. Prod. Rep. New J. Chem. Org. Biomol. Chem. Org. Chem. Front. PHOTOCH PHOTOBIO SCI PCCP Polym. Chem.
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
0
微信
客服QQ
Book学术公众号 扫码关注我们
反馈
×
意见反馈
请填写您的意见或建议
请填写您的手机或邮箱
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
现在去查看 取消
×
提示
确定
Book学术官方微信
Book学术文献互助
Book学术文献互助群
群 号:481959085
Book学术
文献互助 智能选刊 最新文献 互助须知 联系我们:info@booksci.cn
Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。
Copyright © 2023 Book学术 All rights reserved.
ghs 京公网安备 11010802042870号 京ICP备2023020795号-1