This study explored two aspects of temporality in students’ SRL behaviours to understand the dynamics of SRL phase transitions. In the first aspect, which refers to the temporal order of Self-regulated learning (SRL) phases, we characterized four types of SRL processes based on phase transitions and the cyclical nature of SRL. The SRL types were mapped into the kinds of iterative behaviours over SRL phases which correspond to the theorized self-regulatory behaviours of students at different levels of SRL skills. We found a significant association between SRL types and the assignment grades that suggests the higher achieved learning outcomes, i.e., programming skills demonstrated in the assignments, being associated with more advanced SRL processes. This study also focused on the second aspect of temporality, which refers to the instance of time. We revealed the temporal dynamics between SRL phase transitions by analyzing time profiles for transitions in each SRL process type. Next, we showed that a two-day interval is a threshold by which most students iteratively transition from adapting to enactment phases, which provides a suitable time to intervene if the transition is not observed.
{"title":"When to Intervene? Utilizing Two Facets of Temporality in Students’ SRL Processes in a Programming Course","authors":"Sina Nazeri, M. Hatala, Fatemeh Salehian Kia","doi":"10.1145/3576050.3576095","DOIUrl":"https://doi.org/10.1145/3576050.3576095","url":null,"abstract":"This study explored two aspects of temporality in students’ SRL behaviours to understand the dynamics of SRL phase transitions. In the first aspect, which refers to the temporal order of Self-regulated learning (SRL) phases, we characterized four types of SRL processes based on phase transitions and the cyclical nature of SRL. The SRL types were mapped into the kinds of iterative behaviours over SRL phases which correspond to the theorized self-regulatory behaviours of students at different levels of SRL skills. We found a significant association between SRL types and the assignment grades that suggests the higher achieved learning outcomes, i.e., programming skills demonstrated in the assignments, being associated with more advanced SRL processes. This study also focused on the second aspect of temporality, which refers to the instance of time. We revealed the temporal dynamics between SRL phase transitions by analyzing time profiles for transitions in each SRL process type. Next, we showed that a two-day interval is a threshold by which most students iteratively transition from adapting to enactment phases, which provides a suitable time to intervene if the transition is not observed.","PeriodicalId":394433,"journal":{"name":"LAK23: 13th International Learning Analytics and Knowledge Conference","volume":"17 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-03-13","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"115477687","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
This preliminary study focuses on using voice activity detection (VAD) algorithms to extract turn information of small group work detected from recorded individual audio stream data from undergraduate engineering discussion sections. Video data along with audio were manually coded for collaborative behavior of students and teacher-student interaction. We found that individual audio data can be used to obtain features that can describe group work in noisy classrooms. We observed patterns in student turn taking and talk duration during various sections of the classroom which matched with the video coded data. Results show that high quality individual audio data can be effective in describing collaborative processes that occurs in the classroom. Future directions on using prosodic features and implications on how we can conceptualize collaborative group work using audio data are discussed.
{"title":"Turn-taking analysis of small group collaboration in an engineering discussion classroom","authors":"Robin Jephthah Rajarathinam, C. D'Angelo","doi":"10.1145/3576050.3576099","DOIUrl":"https://doi.org/10.1145/3576050.3576099","url":null,"abstract":"This preliminary study focuses on using voice activity detection (VAD) algorithms to extract turn information of small group work detected from recorded individual audio stream data from undergraduate engineering discussion sections. Video data along with audio were manually coded for collaborative behavior of students and teacher-student interaction. We found that individual audio data can be used to obtain features that can describe group work in noisy classrooms. We observed patterns in student turn taking and talk duration during various sections of the classroom which matched with the video coded data. Results show that high quality individual audio data can be effective in describing collaborative processes that occurs in the classroom. Future directions on using prosodic features and implications on how we can conceptualize collaborative group work using audio data are discussed.","PeriodicalId":394433,"journal":{"name":"LAK23: 13th International Learning Analytics and Knowledge Conference","volume":"9 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-03-13","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"116463341","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Daniel Biedermann, J. Schneider, George-Petru Ciordas-Hertel, B. Eichmann, Carolin Hahnel, Frank Goldhammer, H. Drachsler
When reading long and complex texts, students may disengage and miss out on relevant content. In order to prevent disengaged behavior or to counteract it by means of an intervention, it is ideally detected an early stage. In this paper, we present a method for early disengagement detection that relies only on the classification of scrolling data. The presented method transforms scrolling data into a time series representation, where each point of the series represents the vertical position of the viewport in the text document. This time series representation is then classified using time series classification algorithms. We evaluated the method on a dataset of 565 university students reading eight different texts. We compared the algorithm performance with different time series lengths, data sampling strategies, the texts that make up the training data, and classification algorithms. The method can classify disengagement early with up to 70% accuracy. However, we also observe differences in the performance depending on which of the texts are included in the training dataset. We discuss our results and propose several possible improvements to enhance the method.
{"title":"Detecting the Disengaged Reader - Using Scrolling Data to Predict Disengagement during Reading","authors":"Daniel Biedermann, J. Schneider, George-Petru Ciordas-Hertel, B. Eichmann, Carolin Hahnel, Frank Goldhammer, H. Drachsler","doi":"10.1145/3576050.3576078","DOIUrl":"https://doi.org/10.1145/3576050.3576078","url":null,"abstract":"When reading long and complex texts, students may disengage and miss out on relevant content. In order to prevent disengaged behavior or to counteract it by means of an intervention, it is ideally detected an early stage. In this paper, we present a method for early disengagement detection that relies only on the classification of scrolling data. The presented method transforms scrolling data into a time series representation, where each point of the series represents the vertical position of the viewport in the text document. This time series representation is then classified using time series classification algorithms. We evaluated the method on a dataset of 565 university students reading eight different texts. We compared the algorithm performance with different time series lengths, data sampling strategies, the texts that make up the training data, and classification algorithms. The method can classify disengagement early with up to 70% accuracy. However, we also observe differences in the performance depending on which of the texts are included in the training dataset. We discuss our results and propose several possible improvements to enhance the method.","PeriodicalId":394433,"journal":{"name":"LAK23: 13th International Learning Analytics and Knowledge Conference","volume":"15 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-03-13","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"127180900","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Jionghao Lin, Wei Dai, Lisa-Angelique Lim, Yi-Shan Tsai, R. F. Mello, Hassan Khosravi, D. Gašević, Guanliang Chen
Feedback is an effective way to assist students in achieving learning goals. The conceptualisation of feedback is gradually moving from feedback as information to feedback as a learner-centred process. To demonstrate feedback effectiveness, feedback as a learner-centred process should be designed to provide quality feedback content and promote student learning outcomes on the subsequent task. However, it remains unclear how instructors adopt the learner-centred feedback framework for feedback provision in the teaching practice. Thus, our study made use of a comprehensive learner-centred feedback framework to analyse feedback content and identify the characteristics of feedback content among student groups with different performance changes. Specifically, we collected the instructors’ feedback on two consecutive assignments offered by an introductory to data science course at the postgraduate level. On the basis of the first assignment, we used the status of student grade changes (i.e., students whose performance increased and those whose performance did not increase on the second assignment) as the proxy of the student learning outcomes. Then, we engineered and extracted features from the feedback content on the first assignment using a learner-centred feedback framework and further examined the differences of these features between different groups of student learning outcomes. Lastly, we used the features to predict student learning outcomes by using widely-used machine learning models and provided the interpretation of predicted results by using the SHapley Additive exPlanations (SHAP) framework. We found that 1) most features from the feedback content presented significant differences between the groups of student learning outcomes, 2) the gradient boost tree model could effectively predict student learning outcomes, and 3) SHAP could transparently interpret the feature importance on predictions.
{"title":"Learner-centred Analytics of Feedback Content in Higher Education","authors":"Jionghao Lin, Wei Dai, Lisa-Angelique Lim, Yi-Shan Tsai, R. F. Mello, Hassan Khosravi, D. Gašević, Guanliang Chen","doi":"10.1145/3576050.3576064","DOIUrl":"https://doi.org/10.1145/3576050.3576064","url":null,"abstract":"Feedback is an effective way to assist students in achieving learning goals. The conceptualisation of feedback is gradually moving from feedback as information to feedback as a learner-centred process. To demonstrate feedback effectiveness, feedback as a learner-centred process should be designed to provide quality feedback content and promote student learning outcomes on the subsequent task. However, it remains unclear how instructors adopt the learner-centred feedback framework for feedback provision in the teaching practice. Thus, our study made use of a comprehensive learner-centred feedback framework to analyse feedback content and identify the characteristics of feedback content among student groups with different performance changes. Specifically, we collected the instructors’ feedback on two consecutive assignments offered by an introductory to data science course at the postgraduate level. On the basis of the first assignment, we used the status of student grade changes (i.e., students whose performance increased and those whose performance did not increase on the second assignment) as the proxy of the student learning outcomes. Then, we engineered and extracted features from the feedback content on the first assignment using a learner-centred feedback framework and further examined the differences of these features between different groups of student learning outcomes. Lastly, we used the features to predict student learning outcomes by using widely-used machine learning models and provided the interpretation of predicted results by using the SHapley Additive exPlanations (SHAP) framework. We found that 1) most features from the feedback content presented significant differences between the groups of student learning outcomes, 2) the gradient boost tree model could effectively predict student learning outcomes, and 3) SHAP could transparently interpret the feature importance on predictions.","PeriodicalId":394433,"journal":{"name":"LAK23: 13th International Learning Analytics and Knowledge Conference","volume":"16 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-03-13","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"123939230","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
F. M. Fahid, S. J. Lee, Bradford W. Mott, Jessica Vandenberg, Halim Acosta, T. Brush, Krista D. Glazewski, C. Hmelo‐Silver, James Lester
Collaborative game-based learning environments have significant potential for creating effective and engaging group learning experiences. These environments offer rich interactions between small groups of students by embedding collaborative problem solving within immersive virtual worlds. Students often share information, ask questions, negotiate, and construct explanations between themselves towards solving a common goal. However, students sometimes disengage from the learning activities, and due to the nature of collaboration, their disengagement can propagate and negatively impact others within the group. From a teacher's perspective, it can be challenging to identify disengaged students within different groups in a classroom as they need to spend a significant amount of time orchestrating the classroom. Prior work has explored automated frameworks for identifying behavioral disengagement. However, most prior work relies on a single modality for identifying disengagement. In this work, we investigate the effects of using multiple modalities to detect disengagement behaviors of students in a collaborative game-based learning environment. For that, we utilized facial video recordings and group chat messages of 26 middle school students while they were interacting with Crystal Island: EcoJourneys, a game-based learning environment for ecosystem science. Our study shows that the predictive accuracy of a unimodal model heavily relies on the modality of the ground truth, whereas multimodal models surpass the unimodal models, trading resources for accuracy. Our findings can benefit future researchers in designing behavioral engagement detection frameworks for assisting teachers in using collaborative game-based learning within their classrooms.
{"title":"Effects of Modalities in Detecting Behavioral Engagement in Collaborative Game-Based Learning","authors":"F. M. Fahid, S. J. Lee, Bradford W. Mott, Jessica Vandenberg, Halim Acosta, T. Brush, Krista D. Glazewski, C. Hmelo‐Silver, James Lester","doi":"10.1145/3576050.3576079","DOIUrl":"https://doi.org/10.1145/3576050.3576079","url":null,"abstract":"Collaborative game-based learning environments have significant potential for creating effective and engaging group learning experiences. These environments offer rich interactions between small groups of students by embedding collaborative problem solving within immersive virtual worlds. Students often share information, ask questions, negotiate, and construct explanations between themselves towards solving a common goal. However, students sometimes disengage from the learning activities, and due to the nature of collaboration, their disengagement can propagate and negatively impact others within the group. From a teacher's perspective, it can be challenging to identify disengaged students within different groups in a classroom as they need to spend a significant amount of time orchestrating the classroom. Prior work has explored automated frameworks for identifying behavioral disengagement. However, most prior work relies on a single modality for identifying disengagement. In this work, we investigate the effects of using multiple modalities to detect disengagement behaviors of students in a collaborative game-based learning environment. For that, we utilized facial video recordings and group chat messages of 26 middle school students while they were interacting with Crystal Island: EcoJourneys, a game-based learning environment for ecosystem science. Our study shows that the predictive accuracy of a unimodal model heavily relies on the modality of the ground truth, whereas multimodal models surpass the unimodal models, trading resources for accuracy. Our findings can benefit future researchers in designing behavioral engagement detection frameworks for assisting teachers in using collaborative game-based learning within their classrooms.","PeriodicalId":394433,"journal":{"name":"LAK23: 13th International Learning Analytics and Knowledge Conference","volume":"38 4","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-03-13","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"133111357","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pankaj Chejara, L. Prieto, M. Rodríguez-Triana, Reet Kasepalu, Adolfo Ruiz-Calleja, Shashi Kant Shankar
Multimodal learning analytics (MMLA) research for building collaboration quality estimation models has shown significant progress. However, the generalizability of such models is seldom addressed. In this paper, we address this gap by systematically evaluating the across-context generalizability of collaboration quality models developed using a typical MMLA pipeline. This paper further presents a methodology to explore modelling pipelines with different configurations to improve the generalizability of the model. We collected 11 multimodal datasets (audio and log data) from face-to-face collaborative learning activities in six different classrooms with five different subject teachers. Our results showed that the models developed using the often-employed MMLA pipeline degraded in terms of Kappa from Fair (.20 < Kappa < .40) to Poor (Kappa < .20) when evaluated across contexts. This degradation in performance was significantly ameliorated with pipelines that emerged as high-performing from our exploration of 32 pipelines. Furthermore, our exploration of pipelines provided statistical evidence that often-overlooked contextual data features improve the generalizability of a collaboration quality model. With these findings, we make recommendations for the modelling pipeline which can potentially help other researchers in achieving better generalizability in their collaboration quality estimation models.
{"title":"How to Build More Generalizable Models for Collaboration Quality? Lessons Learned from Exploring Multi-Context Audio-Log Datasets using Multimodal Learning Analytics","authors":"Pankaj Chejara, L. Prieto, M. Rodríguez-Triana, Reet Kasepalu, Adolfo Ruiz-Calleja, Shashi Kant Shankar","doi":"10.1145/3576050.3576144","DOIUrl":"https://doi.org/10.1145/3576050.3576144","url":null,"abstract":"Multimodal learning analytics (MMLA) research for building collaboration quality estimation models has shown significant progress. However, the generalizability of such models is seldom addressed. In this paper, we address this gap by systematically evaluating the across-context generalizability of collaboration quality models developed using a typical MMLA pipeline. This paper further presents a methodology to explore modelling pipelines with different configurations to improve the generalizability of the model. We collected 11 multimodal datasets (audio and log data) from face-to-face collaborative learning activities in six different classrooms with five different subject teachers. Our results showed that the models developed using the often-employed MMLA pipeline degraded in terms of Kappa from Fair (.20 < Kappa < .40) to Poor (Kappa < .20) when evaluated across contexts. This degradation in performance was significantly ameliorated with pipelines that emerged as high-performing from our exploration of 32 pipelines. Furthermore, our exploration of pipelines provided statistical evidence that often-overlooked contextual data features improve the generalizability of a collaboration quality model. With these findings, we make recommendations for the modelling pipeline which can potentially help other researchers in achieving better generalizability in their collaboration quality estimation models.","PeriodicalId":394433,"journal":{"name":"LAK23: 13th International Learning Analytics and Knowledge Conference","volume":"9 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-03-13","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"114957855","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
R. Moulder, Brandon M. Booth, Angelina Abitino, Sidney K. D’Mello
Shared visual attention between team members facilitates collaborative problem solving (CPS), but little is known about how team-level eye gaze dynamics influence the quality and successfulness of CPS. To better understand the role of shared visual attention during CPS, we collected eye gaze data from 279 individuals solving computer-based physics puzzles while in teams of three. We converted eye gaze into discrete screen locations and quantified team-level gaze dynamics using recurrence quantification analysis (RQA). Specifically, we used a centroid-based auto-RQA approach, a pairwise team member cross-RQAs approach, and a multi-dimensional RQA approach to quantify team-level eye gaze dynamics from the eye gaze data of team members. We find that teams differing in composition based on prior task knowledge, gender, and race show few differences in team-level eye gaze dynamics. We also find that RQA metrics of team-level eye gaze dynamics were predictive of task success (all ps < .001). However, the same metrics showed different patterns of feature importance depending on predictive model and RQA type, suggesting some redundancy in task-relevant information. These findings signify that team-level eye gaze dynamics play an important role in CPS and that different forms of RQA pick up on unique aspects of shared attention between team-members.
{"title":"Recurrence Quantification Analysis of Eye Gaze Dynamics During Team Collaboration","authors":"R. Moulder, Brandon M. Booth, Angelina Abitino, Sidney K. D’Mello","doi":"10.1145/3576050.3576113","DOIUrl":"https://doi.org/10.1145/3576050.3576113","url":null,"abstract":"Shared visual attention between team members facilitates collaborative problem solving (CPS), but little is known about how team-level eye gaze dynamics influence the quality and successfulness of CPS. To better understand the role of shared visual attention during CPS, we collected eye gaze data from 279 individuals solving computer-based physics puzzles while in teams of three. We converted eye gaze into discrete screen locations and quantified team-level gaze dynamics using recurrence quantification analysis (RQA). Specifically, we used a centroid-based auto-RQA approach, a pairwise team member cross-RQAs approach, and a multi-dimensional RQA approach to quantify team-level eye gaze dynamics from the eye gaze data of team members. We find that teams differing in composition based on prior task knowledge, gender, and race show few differences in team-level eye gaze dynamics. We also find that RQA metrics of team-level eye gaze dynamics were predictive of task success (all ps < .001). However, the same metrics showed different patterns of feature importance depending on predictive model and RQA type, suggesting some redundancy in task-relevant information. These findings signify that team-level eye gaze dynamics play an important role in CPS and that different forms of RQA pick up on unique aspects of shared attention between team-members.","PeriodicalId":394433,"journal":{"name":"LAK23: 13th International Learning Analytics and Knowledge Conference","volume":"1 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-03-13","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"130729247","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Xinyu Li, Lixiang Yan, Linxuan Zhao, Roberto Martínez-Maldonado, D. Gašević
Capturing data on socio-spatial behaviours is essential in obtaining meaningful educational insights into collaborative learning and teamwork in co-located learning contexts. Existing solutions, however, have limitations regarding scalability and practicality since they rely largely on costly location tracking systems, are labour-intensive, or are unsuitable for complex learning environments. To address these limitations, we propose an innovative computer-vision-based approach – Computer Vision for Position Estimation (CVPE) – for collecting socio-spatial data in complex learning settings where sophisticated collaborations occur. CVPE is scalable and practical with a fast processing time and only needs low-cost hardware (e.g., cameras and computers). The built-in privacy protection modules also minimise potential privacy and data security issues by masking individuals’ facial identities and provide options to automatically delete recordings after processing, making CVPE a suitable option for generating continuous multimodal/classroom analytics. The potential of CVPE was evaluated by applying it to analyse video data about teamwork in simulation-based learning. The results showed that CVPE extracted socio-spatial behaviours relatively reliably from video recordings compared to indoor positioning data. These socio-spatial behaviours extracted with CVPE uncovered valuable insights into teamwork when analysed with epistemic network analysis. The limitations of CVPE for effective use in learning analytics are also discussed.
{"title":"CVPE: A Computer Vision Approach for Scalable and Privacy-Preserving Socio-spatial, Multimodal Learning Analytics","authors":"Xinyu Li, Lixiang Yan, Linxuan Zhao, Roberto Martínez-Maldonado, D. Gašević","doi":"10.1145/3576050.3576145","DOIUrl":"https://doi.org/10.1145/3576050.3576145","url":null,"abstract":"Capturing data on socio-spatial behaviours is essential in obtaining meaningful educational insights into collaborative learning and teamwork in co-located learning contexts. Existing solutions, however, have limitations regarding scalability and practicality since they rely largely on costly location tracking systems, are labour-intensive, or are unsuitable for complex learning environments. To address these limitations, we propose an innovative computer-vision-based approach – Computer Vision for Position Estimation (CVPE) – for collecting socio-spatial data in complex learning settings where sophisticated collaborations occur. CVPE is scalable and practical with a fast processing time and only needs low-cost hardware (e.g., cameras and computers). The built-in privacy protection modules also minimise potential privacy and data security issues by masking individuals’ facial identities and provide options to automatically delete recordings after processing, making CVPE a suitable option for generating continuous multimodal/classroom analytics. The potential of CVPE was evaluated by applying it to analyse video data about teamwork in simulation-based learning. The results showed that CVPE extracted socio-spatial behaviours relatively reliably from video recordings compared to indoor positioning data. These socio-spatial behaviours extracted with CVPE uncovered valuable insights into teamwork when analysed with epistemic network analysis. The limitations of CVPE for effective use in learning analytics are also discussed.","PeriodicalId":394433,"journal":{"name":"LAK23: 13th International Learning Analytics and Knowledge Conference","volume":"60 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-03-13","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"122265755","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Wemerson Marinho, E. W. Clua, Luis Martí, Karla Marinho
Recently, new Neural Language Models pre-trained on a massive corpus of texts are available. These models encode statistical features of the languages through their parameters, creating better word vector representations that allow the training of neural networks with smaller sample sets. In this context, we investigate the application of these models to predict Item Response Theory parameters in multiple choice questions. More specifically, we apply our models for the Brazilian National High School Exam (ENEM) questions using the text of their statements and propose a novel optimization target for regression: Item Characteristic Curve. The architecture employed could predict the difficulty parameter b of the ENEM 2020 and 2021 items with a mean absolute error of 70 points. Calculating the IRT score in each knowledge area of the exam for a sample of 100,000 students, we obtained a mean absolute below 40 points for all knowledge areas. Considering only the top quartile, the exam’s main target of interest, the average error was less than 30 points for all areas, being the majority lower than 15 points. Such performance allows predicting parameters on newly created questions, composing mock tests for student training, and analyzing their performance with excellent precision, dispensing with the need for costly item calibration pre-test step.
{"title":"Predicting Item Response Theory Parameters Using Question Statements Texts","authors":"Wemerson Marinho, E. W. Clua, Luis Martí, Karla Marinho","doi":"10.1145/3576050.3576139","DOIUrl":"https://doi.org/10.1145/3576050.3576139","url":null,"abstract":"Recently, new Neural Language Models pre-trained on a massive corpus of texts are available. These models encode statistical features of the languages through their parameters, creating better word vector representations that allow the training of neural networks with smaller sample sets. In this context, we investigate the application of these models to predict Item Response Theory parameters in multiple choice questions. More specifically, we apply our models for the Brazilian National High School Exam (ENEM) questions using the text of their statements and propose a novel optimization target for regression: Item Characteristic Curve. The architecture employed could predict the difficulty parameter b of the ENEM 2020 and 2021 items with a mean absolute error of 70 points. Calculating the IRT score in each knowledge area of the exam for a sample of 100,000 students, we obtained a mean absolute below 40 points for all knowledge areas. Considering only the top quartile, the exam’s main target of interest, the average error was less than 30 points for all areas, being the majority lower than 15 points. Such performance allows predicting parameters on newly created questions, composing mock tests for student training, and analyzing their performance with excellent precision, dispensing with the need for costly item calibration pre-test step.","PeriodicalId":394433,"journal":{"name":"LAK23: 13th International Learning Analytics and Knowledge Conference","volume":"42 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-03-13","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"127662414","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Student engagement has been identified as a critical construct for understanding and predicting educational success. However, research has shown that it can be hard to align data-driven insights of engagement with observed and self-reported levels of engagement. Given the emergence and increasing application of learning analytics (LA) within K-12 education, further research is needed to understand how engagement is being conceptualized and measured within LA research. This scoping review identifies and synthesizes literature published between 2011-2022, focused on LA and student engagement in K-12 contexts, and indexed in five international databases. 27 articles and conference papers from 13 different countries were included for review. We found that most of the research was undertaken in middle school years within STEM subjects. The results show that there is a wide discrepancy in researchers’ understanding and operationalization of engagement and little evidence to suggest that LA improves learning outcomes and support. However, the potential to do so remains strong. Guidance is provided for future LA engagement research to better align with these goals.
{"title":"The current state of using learning analytics to measure and support K-12 student engagement: A scoping review","authors":"Melissa Bond, Olga Viberg, Nina Bergdahl","doi":"10.1145/3576050.3576085","DOIUrl":"https://doi.org/10.1145/3576050.3576085","url":null,"abstract":"Student engagement has been identified as a critical construct for understanding and predicting educational success. However, research has shown that it can be hard to align data-driven insights of engagement with observed and self-reported levels of engagement. Given the emergence and increasing application of learning analytics (LA) within K-12 education, further research is needed to understand how engagement is being conceptualized and measured within LA research. This scoping review identifies and synthesizes literature published between 2011-2022, focused on LA and student engagement in K-12 contexts, and indexed in five international databases. 27 articles and conference papers from 13 different countries were included for review. We found that most of the research was undertaken in middle school years within STEM subjects. The results show that there is a wide discrepancy in researchers’ understanding and operationalization of engagement and little evidence to suggest that LA improves learning outcomes and support. However, the potential to do so remains strong. Guidance is provided for future LA engagement research to better align with these goals.","PeriodicalId":394433,"journal":{"name":"LAK23: 13th International Learning Analytics and Knowledge Conference","volume":"11 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-03-13","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"125953363","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
京公网安备 11010802042870号
京ICP备2023020795号-1
扫码关注我们
求助内容：
应助结果提醒方式：