An engagement-aware predictive model to evaluate problem-solving performance from the study of adult skills' (PIAAC 2012) process data

IF 2.6 Q1 EDUCATION & EDUCATIONAL RESEARCH Large-Scale Assessments in Education Pub Date : 2024-03-01 DOI:10.1186/s40536-024-00194-y

Jinnie Shin, Bowen Wang, Wallace N. Pinto Junior, Mark J. Gierl

{"title":"An engagement-aware predictive model to evaluate problem-solving performance from the study of adult skills' (PIAAC 2012) process data","authors":"Jinnie Shin, Bowen Wang, Wallace N. Pinto Junior, Mark J. Gierl","doi":"10.1186/s40536-024-00194-y","DOIUrl":null,"url":null,"abstract":"<h3>Abstract</h3> <p>The benefits of incorporating process information in a large-scale assessment with the complex micro-level evidence from the examinees (i.e., process log data) are well documented in the research across large-scale assessments and learning analytics. This study introduces a deep-learning-based approach to predictive modeling of the examinee’s performance in sequential, interactive problem-solving tasks from a large-scale assessment of adults' educational competencies. The current methods disambiguate problem-solving behaviors using network analysis to inform the examinee's performance in a series of problem-solving tasks. The unique contribution of this framework lies in the introduction of an “effort-aware” system. The system considers the information regarding the examinee’s task-engagement level to accurately predict their task performance. The study demonstrates the potential to introduce a high-performing deep learning model to learning analytics and examinee performance modeling in a large-scale problem-solving task environment collected from the OECD Programme for the International Assessment of Adult Competencies (PIAAC 2012) test in multiple countries, including the United States, South Korea, and the United Kingdom. Our findings indicated a close relationship between the examinee's engagement level and their problem-solving skills as well as the importance of modeling them together to have a better measure of students’ problem-solving performance.</p>","PeriodicalId":37009,"journal":{"name":"Large-Scale Assessments in Education","volume":"6 1","pages":""},"PeriodicalIF":2.6000,"publicationDate":"2024-03-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Large-Scale Assessments in Education","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/10.1186/s40536-024-00194-y","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"EDUCATION & EDUCATIONAL RESEARCH","Score":null,"Total":0}

引用次数: 0

Abstract

The benefits of incorporating process information in a large-scale assessment with the complex micro-level evidence from the examinees (i.e., process log data) are well documented in the research across large-scale assessments and learning analytics. This study introduces a deep-learning-based approach to predictive modeling of the examinee’s performance in sequential, interactive problem-solving tasks from a large-scale assessment of adults' educational competencies. The current methods disambiguate problem-solving behaviors using network analysis to inform the examinee's performance in a series of problem-solving tasks. The unique contribution of this framework lies in the introduction of an “effort-aware” system. The system considers the information regarding the examinee’s task-engagement level to accurately predict their task performance. The study demonstrates the potential to introduce a high-performing deep learning model to learning analytics and examinee performance modeling in a large-scale problem-solving task environment collected from the OECD Programme for the International Assessment of Adult Competencies (PIAAC 2012) test in multiple countries, including the United States, South Korea, and the United Kingdom. Our findings indicated a close relationship between the examinee's engagement level and their problem-solving skills as well as the importance of modeling them together to have a better measure of students’ problem-solving performance.

查看原文

微信好友朋友圈 QQ好友复制链接

本刊更多论文

从成人技能研究（PIAAC 2012）过程数据中建立参与意识预测模型，以评估解决问题的绩效

摘要在大规模评估和学习分析的研究中，将过程信息与来自受测者的复杂微观证据（即过程日志数据）结合起来的好处已得到充分证明。本研究介绍了一种基于深度学习的方法，用于对成人教育能力大规模测评中受测者在连续、交互式问题解决任务中的表现进行预测建模。目前的方法利用网络分析法对问题解决行为进行了区分，从而为受测者在一系列问题解决任务中的表现提供信息。该框架的独特之处在于引入了 "努力感知 "系统。该系统考虑了考生的任务参与水平信息，以准确预测他们的任务表现。本研究展示了在大规模问题解决任务环境中引入高性能深度学习模型进行学习分析和考生成绩建模的潜力，该环境收集自经合组织成人能力国际评估项目（PIAAC 2012）在美国、韩国和英国等多个国家进行的测试。我们的研究结果表明，考生的参与程度与他们的问题解决能力之间存在密切关系，同时，为了更好地衡量学生的问题解决能力，必须将两者结合起来进行建模。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文去求助

来源期刊