Pub Date : 2024-03-22DOI: 10.1109/TLT.2024.3381028
Lishan Zhang;Linyu Deng;Sixv Zhang;Ling Chen
With the popularity of online one-to-one tutoring, there are emerging concerns about the quality and effectiveness of this kind of tutoring. Although there are some evaluation methods available, they are heavily relied on manual coding by experts, which is too costly. Therefore, using machine learning to predict instruction quality automatically is an effective way to reduce human costs. Three classification methods are analyzed in this article: 1) random forest algorithm with human-engineered descriptive features; 2) long and short-term memory algorithm with acoustic features generated by open speech and music interpretation by large space extraction toolkit; and 3) convolutional neural network algorithm with Mel spectrogram of the audio. The results show that the three approaches can complete the prediction task well, with the second approach exhibiting the best accuracy. The importance of the features in these classification models is analyzed according to eXplainable Artificial Intelligence techniques (i.e., XAI) and statistical feature analysis methods. In this way, key indicators of high-quality tutoring are identified. This study demonstrated the usefulness of XAI techniques in understanding why some tutoring sessions are of good quality and others are not. The results can be potentially used to guide the improvement of online one-to-one tutoring in the future.
{"title":"How Well Can Tutoring Audio Be Autoclassified and Machine Explained With XAI: A Comparison of Three Types of Methods","authors":"Lishan Zhang;Linyu Deng;Sixv Zhang;Ling Chen","doi":"10.1109/TLT.2024.3381028","DOIUrl":"10.1109/TLT.2024.3381028","url":null,"abstract":"With the popularity of online one-to-one tutoring, there are emerging concerns about the quality and effectiveness of this kind of tutoring. Although there are some evaluation methods available, they are heavily relied on manual coding by experts, which is too costly. Therefore, using machine learning to predict instruction quality automatically is an effective way to reduce human costs. Three classification methods are analyzed in this article: 1) random forest algorithm with human-engineered descriptive features; 2) long and short-term memory algorithm with acoustic features generated by open speech and music interpretation by large space extraction toolkit; and 3) convolutional neural network algorithm with Mel spectrogram of the audio. The results show that the three approaches can complete the prediction task well, with the second approach exhibiting the best accuracy. The importance of the features in these classification models is analyzed according to eXplainable Artificial Intelligence techniques (i.e., XAI) and statistical feature analysis methods. In this way, key indicators of high-quality tutoring are identified. This study demonstrated the usefulness of XAI techniques in understanding why some tutoring sessions are of good quality and others are not. The results can be potentially used to guide the improvement of online one-to-one tutoring in the future.","PeriodicalId":49191,"journal":{"name":"IEEE Transactions on Learning Technologies","volume":"17 ","pages":"1302-1312"},"PeriodicalIF":3.7,"publicationDate":"2024-03-22","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"140198862","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"教育学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-03-21DOI: 10.1109/TLT.2024.3403710
Han Wan;Hongzhen Luo;Mengying Li;Xiaoyan Luo
Automatic program repair (APR) tools are valuable for students to assist them with debugging tasks since program repair captures the code modification to make a buggy program pass the given test-suite. However, the process of manually generating catalogs of code modifications is intricate and time-consuming. This article proposes contextual error model repair (CEMR), an automated program repair tool for introductory programming assignments. CEMR is designed to learn program code modifications from incorrect–correct code pairs automatically. Then, it utilizes these code modifications along with CodeBERT, a generative AI, to repair students' new incorrect programs in the same programming assignment. CEMR builds on the observation that code edits performed by students in pairs of incorrect–correct code can be used as input–output examples for learning code modifications. The key idea of CEMR is to leverage the wisdom of the crowd : it uses the existing code modifications of incorrect–correct student code pairs to repair the new incorrect student attempts. We chose three of the most related APR tools, Refazer, Refactory, and AlphaRepair, as the baselines to compare against CEMR. The experimental results demonstrate that, on public and real classroom datasets, CEMR achieves higher repair rates than the baselines. Through further analysis, CEMR has demonstrated promising effectiveness in addressing semantical and logical errors while its performance in fixing syntactical errors is limited. In terms of time for repairing buggy programs, CEMR costs approximately half as much as AlphaRepair requires. We opine that CEMR not only be seen as a program repair method that achieves good results with incorrect–correct code pairs but also be further utilized to generate hints to better assist students in learning programming.
{"title":"Automated Program Repair for Introductory Programming Assignments","authors":"Han Wan;Hongzhen Luo;Mengying Li;Xiaoyan Luo","doi":"10.1109/TLT.2024.3403710","DOIUrl":"10.1109/TLT.2024.3403710","url":null,"abstract":"Automatic program repair (APR) tools are valuable for students to assist them with debugging tasks since program repair captures the code modification to make a buggy program pass the given test-suite. However, the process of manually generating catalogs of code modifications is intricate and time-consuming. This article proposes contextual error model repair (CEMR), an automated program repair tool for introductory programming assignments. CEMR is designed to learn program code modifications from incorrect–correct code pairs automatically. Then, it utilizes these code modifications along with CodeBERT, a generative AI, to repair students' new incorrect programs in the same programming assignment. CEMR builds on the observation that code edits performed by students in pairs of incorrect–correct code can be used as input–output examples for learning code modifications. The key idea of CEMR is to leverage the \u0000<italic>wisdom of the crowd</i>\u0000: it uses the existing code modifications of incorrect–correct student code pairs to repair the new incorrect student attempts. We chose three of the most related APR tools, Refazer, Refactory, and AlphaRepair, as the baselines to compare against CEMR. The experimental results demonstrate that, on public and real classroom datasets, CEMR achieves higher repair rates than the baselines. Through further analysis, CEMR has demonstrated promising effectiveness in addressing semantical and logical errors while its performance in fixing syntactical errors is limited. In terms of time for repairing buggy programs, CEMR costs approximately half as much as AlphaRepair requires. We opine that CEMR not only be seen as a program repair method that achieves good results with incorrect–correct code pairs but also be further utilized to generate hints to better assist students in learning programming.","PeriodicalId":49191,"journal":{"name":"IEEE Transactions on Learning Technologies","volume":"17 ","pages":"1745-1760"},"PeriodicalIF":3.7,"publicationDate":"2024-03-21","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141146079","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"教育学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-03-20DOI: 10.1109/TLT.2024.3403135
Yu Lu;Deliang Wang;Penghe Chen;Zhi Zhang
Amid the rapid evolution of artificial intelligence (AI), the intricate model structures and opaque decision-making processes of AI-based systems have raised the trustworthy issues in education. We, therefore, first propose a novel three-layer knowledge tracing model designed to address trustworthiness for an intelligent tutoring system. Each layer is crafted to tackle a specific challenge: transparency, explainability, and accountability. We have introduced an explainable AI (xAI) approach to offer technical interpreting information, validated by the established educational theories and principles. The validated interpreting information is subsequently transitioned from its technical context into educational insights, which are then incorporated into the newly designed user interface. Our evaluations indicate that an intelligent tutoring system, when equipped with the designed trustworthy knowledge tracing model, significantly enhances user trust and knowledge from the perspectives of both teachers and students. This study, thus, contributes a tangible solution that utilizes the xAI approach as the enabling technology to construct trustworthy systems or tools in education.
在人工智能（AI）飞速发展的过程中，基于 AI 的系统错综复杂的模型结构和不透明的决策过程引发了教育领域的可信性问题。因此，我们首先提出了一个新颖的三层知识追踪模型，旨在解决智能辅导系统的可信性问题。每一层的设计都是为了应对一个特定的挑战：透明度、可解释性和问责制。我们引入了一种可解释的人工智能（xAI）方法来提供技术解释信息，并通过既定的教育理论和原则进行验证。经过验证的解释信息随后会从其技术背景转换为教育见解，然后将其纳入新设计的用户界面。我们的评估结果表明，当智能辅导系统配备所设计的可信知识追踪模型时，从教师和学生的角度来看，都能显著增强用户的信任度和知识水平。因此，这项研究提供了一个切实可行的解决方案，利用 xAI 方法作为构建教育可信系统或工具的使能技术。
{"title":"Design and Evaluation of Trustworthy Knowledge Tracing Model for Intelligent Tutoring System","authors":"Yu Lu;Deliang Wang;Penghe Chen;Zhi Zhang","doi":"10.1109/TLT.2024.3403135","DOIUrl":"10.1109/TLT.2024.3403135","url":null,"abstract":"Amid the rapid evolution of artificial intelligence (AI), the intricate model structures and opaque decision-making processes of AI-based systems have raised the trustworthy issues in education. We, therefore, first propose a novel three-layer knowledge tracing model designed to address trustworthiness for an intelligent tutoring system. Each layer is crafted to tackle a specific challenge: transparency, explainability, and accountability. We have introduced an explainable AI (xAI) approach to offer technical interpreting information, validated by the established educational theories and principles. The validated interpreting information is subsequently transitioned from its technical context into educational insights, which are then incorporated into the newly designed user interface. Our evaluations indicate that an intelligent tutoring system, when equipped with the designed trustworthy knowledge tracing model, significantly enhances user trust and knowledge from the perspectives of both teachers and students. This study, thus, contributes a tangible solution that utilizes the xAI approach as the enabling technology to construct trustworthy systems or tools in education.","PeriodicalId":49191,"journal":{"name":"IEEE Transactions on Learning Technologies","volume":"17 ","pages":"1701-1716"},"PeriodicalIF":3.7,"publicationDate":"2024-03-20","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141146168","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"教育学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-03-18DOI: 10.1109/TLT.2024.3378306
Shurui Bai;Donn Emmanuel Gonda;Khe Foon Hew
This case study explored the use of generative artificial intelligence (GenAI), specifically chat generative pretraining transformer (ChatGPT), in writing scenarios for scenario-based learning (SBL). Our research addressed three key questions: 1) how do teachers leverage GenAI to write scenarios for SBL purposes? 2) what is the quality of GenAI-generated SBL scenarios and tasks? and 3) how does GenAI-supported SBL affect students’ motivation, learning performance, and learning perceptions? A three-step prompting engineering process (write the prompts, curate the output, and verify the output, WCV) was established during the teacher interaction with GenAI in the scenario writing. Findings revealed that by using the WCV approach, ChatGPT enabled the efficient creation of quality scenarios for SBL purposes in a short timeframe. Moreover, students exhibited increased intrinsic motivation, learning performance, and positive attitudes toward GenAI-supported scenarios. We also suggest guidelines for using the WCV prompt engineering process in scenario writing.
{"title":"Write-Curate-Verify: A Case Study of Leveraging Generative AI for Scenario Writing in Scenario-Based Learning","authors":"Shurui Bai;Donn Emmanuel Gonda;Khe Foon Hew","doi":"10.1109/TLT.2024.3378306","DOIUrl":"10.1109/TLT.2024.3378306","url":null,"abstract":"This case study explored the use of generative artificial intelligence (GenAI), specifically chat generative pretraining transformer (ChatGPT), in writing scenarios for scenario-based learning (SBL). Our research addressed three key questions: 1) how do teachers leverage GenAI to write scenarios for SBL purposes? 2) what is the quality of GenAI-generated SBL scenarios and tasks? and 3) how does GenAI-supported SBL affect students’ motivation, learning performance, and learning perceptions? A three-step prompting engineering process (write the prompts, curate the output, and verify the output, WCV) was established during the teacher interaction with GenAI in the scenario writing. Findings revealed that by using the WCV approach, ChatGPT enabled the efficient creation of quality scenarios for SBL purposes in a short timeframe. Moreover, students exhibited increased intrinsic motivation, learning performance, and positive attitudes toward GenAI-supported scenarios. We also suggest guidelines for using the WCV prompt engineering process in scenario writing.","PeriodicalId":49191,"journal":{"name":"IEEE Transactions on Learning Technologies","volume":"17 ","pages":"1313-1324"},"PeriodicalIF":3.7,"publicationDate":"2024-03-18","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"140170034","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"教育学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-03-18DOI: 10.1109/TLT.2024.3378279
Xu Chen;Di Wu
Generative artificial intelligence (AI) is widely recognized as one of the most influential technologies for the future, having sparked a paradigm shift in scientific research. The field of education has also been greatly impacted by this transformative technology, with researchers exploring the applications of generative AI, particularly ChatGPT, in education. However, existing research primarily focuses on generating text from text, and there remains a relative scarcity of studies on leveraging multimodal generation capabilities to address key challenges in multimodal data supported instruction. In this article, we present a technical framework for generating Tang poetry situational videos, emphasizing the utilization of generative AI to address the need for multimedia teaching resources. Our framework comprises three main modules: textual situational comprehension, image creation, and video generation. Moreover, we have developed a situational video generation system that incorporates various technologies, including text-to-text generation models, text-to-image generation models, image interpolation, text-to-speech synthesis, and video synthesis. To ascertain the efficacy of the modules within the Tang poetry situational video generation system, we undertook a comparative analysis utilizing the prevalent text-to-image and text-to-video generation models. The empirical findings indicate that our approach is capable of generating images that exhibit greater semantic similarity with the poems, thereby enabling a better comprehension of the poem's connotations and its key components. Concurrently, the Tang poetry videos generated can significantly contribute to the reduction of cognitive load and the enhancement of understanding during the learning process. Our research showcases the potential of generative AI in the education field, specifically in the domain of multimodal teaching resources.
{"title":"Automatic Generation of Multimedia Teaching Materials Based on Generative AI: Taking Tang Poetry as an Example","authors":"Xu Chen;Di Wu","doi":"10.1109/TLT.2024.3378279","DOIUrl":"10.1109/TLT.2024.3378279","url":null,"abstract":"Generative artificial intelligence (AI) is widely recognized as one of the most influential technologies for the future, having sparked a paradigm shift in scientific research. The field of education has also been greatly impacted by this transformative technology, with researchers exploring the applications of generative AI, particularly ChatGPT, in education. However, existing research primarily focuses on generating text from text, and there remains a relative scarcity of studies on leveraging multimodal generation capabilities to address key challenges in multimodal data supported instruction. In this article, we present a technical framework for generating Tang poetry situational videos, emphasizing the utilization of generative AI to address the need for multimedia teaching resources. Our framework comprises three main modules: textual situational comprehension, image creation, and video generation. Moreover, we have developed a situational video generation system that incorporates various technologies, including text-to-text generation models, text-to-image generation models, image interpolation, text-to-speech synthesis, and video synthesis. To ascertain the efficacy of the modules within the Tang poetry situational video generation system, we undertook a comparative analysis utilizing the prevalent text-to-image and text-to-video generation models. The empirical findings indicate that our approach is capable of generating images that exhibit greater semantic similarity with the poems, thereby enabling a better comprehension of the poem's connotations and its key components. Concurrently, the Tang poetry videos generated can significantly contribute to the reduction of cognitive load and the enhancement of understanding during the learning process. Our research showcases the potential of generative AI in the education field, specifically in the domain of multimodal teaching resources.","PeriodicalId":49191,"journal":{"name":"IEEE Transactions on Learning Technologies","volume":"17 ","pages":"1353-1366"},"PeriodicalIF":3.7,"publicationDate":"2024-03-18","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"140170039","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"教育学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-03-16DOI: 10.1109/TLT.2024.3401734
Pallavi Singh;Phat K. Huynh;Dang Nguyen;Trung Q. Le;Wilfrido Moreno
In organizational and academic settings, the strategic formation of teams is paramount, necessitating an approach that transcends conventional methodologies. This study introduces a novel application of multicriteria integer programming (MCIP), which simultaneously accommodates multiple criteria, thereby innovatively addressing the complex task of team formation. Unlike traditional single-objective optimization methods, our research designs a comprehensive framework capable of modeling a wide array of factors, including skill levels, backgrounds, and personality traits. The objective function of this framework is optimized to maximize within-team diversity while minimizing both conflict levels and variance in diversity between teams. Central to our approach is a two-stage optimization process. Initially, it segments the population into subgroups using a weighted heterogeneous multivariate K-means algorithm, allowing for a targeted and nuanced team assembly. This is followed by the application of a surrogate optimization technique within these subgroups, efficiently navigating the complexities of MCIP for large-scale applications. Our approach is further enhanced by the inclusion of explicit constraints such as potential interpersonal conflicts, a factor often overlooked in previous studies. The results from our study demonstrate the optimality and robustness of our model across simulation scenarios with different data heterogeneity levels. The contributions of this study are manifold, addressing critical gaps in the existing literature with a theory-backed, empirically validated framework for advanced team formation. Beyond theoretical implications, our work provides a practical guide for implementing conflict-aware, sophisticated team formation strategies in real-world scenarios. This advancement paves the way for future research to explore and enhance this model, providing more sophisticated and efficient team formation strategies.
{"title":"Leveraging Multicriteria Integer Programming Optimization for Effective Team Formation","authors":"Pallavi Singh;Phat K. Huynh;Dang Nguyen;Trung Q. Le;Wilfrido Moreno","doi":"10.1109/TLT.2024.3401734","DOIUrl":"10.1109/TLT.2024.3401734","url":null,"abstract":"In organizational and academic settings, the strategic formation of teams is paramount, necessitating an approach that transcends conventional methodologies. This study introduces a novel application of multicriteria integer programming (MCIP), which simultaneously accommodates multiple criteria, thereby innovatively addressing the complex task of team formation. Unlike traditional single-objective optimization methods, our research designs a comprehensive framework capable of modeling a wide array of factors, including skill levels, backgrounds, and personality traits. The objective function of this framework is optimized to maximize within-team diversity while minimizing both conflict levels and variance in diversity between teams. Central to our approach is a two-stage optimization process. Initially, it segments the population into subgroups using a weighted heterogeneous multivariate <italic>K</i>-means algorithm, allowing for a targeted and nuanced team assembly. This is followed by the application of a surrogate optimization technique within these subgroups, efficiently navigating the complexities of MCIP for large-scale applications. Our approach is further enhanced by the inclusion of explicit constraints such as potential interpersonal conflicts, a factor often overlooked in previous studies. The results from our study demonstrate the optimality and robustness of our model across simulation scenarios with different data heterogeneity levels. The contributions of this study are manifold, addressing critical gaps in the existing literature with a theory-backed, empirically validated framework for advanced team formation. Beyond theoretical implications, our work provides a practical guide for implementing conflict-aware, sophisticated team formation strategies in real-world scenarios. This advancement paves the way for future research to explore and enhance this model, providing more sophisticated and efficient team formation strategies.","PeriodicalId":49191,"journal":{"name":"IEEE Transactions on Learning Technologies","volume":"18 ","pages":"72-84"},"PeriodicalIF":2.9,"publicationDate":"2024-03-16","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141063023","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"教育学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-03-15DOI: 10.1109/TLT.2024.3401457
Gyeong-Geon Lee;Xiaoming Zhai
While ongoing efforts have continuously emphasized the integration of ChatGPT with science teaching and learning, there are limited empirical studies exploring its actual utility in the classroom. This study aims to fill this gap by analyzing the lesson plans developed by 29 pre-service elementary teachers and assessing how they integrated ChatGPT into science learning activities. We first examined how ChatGPT was integrated with the subject domains, teaching methods/strategies, and then evaluated the lesson plans using a generative artificial intelligence (AI)-technological pedagogical and content knowledge (TPACK)-based rubric. We further examined pre-service teachers' perceptions and concerns about integrating ChatGPT into science learning. Results show a diverse number of ChatGPT applications in different science domains—e.g., Biology (9/29), Chemistry (7/29), and Earth Science (7/29). A total of 14 types of teaching methods/strategies were identified in the lesson plans. On average, the pre-service teachers' lesson plans scored high on the modified TPACK-based rubric (M = 3.29; SD = 0.91; on a 1–4 scale), indicating a reasonable envisage of integrating ChatGPT into science learning, particularly in “instructional strategies and ChatGPT” (M = 3.48; SD = 0.99). However, they scored relatively lower on exploiting ChatGPT's functions toward its full potential (M = 3.00; SD = 0.93), compared to other aspects. We also identified several inappropriate use cases of ChatGPT in lesson planning (e.g., as a source of hallucinated Internet material and technically unsupported visual guidance). Pre-service teachers anticipated ChatGPT to afford high-quality questioning, self-directed learning, individualized learning support, and formative assessment. Meanwhile, they also expressed concerns about its accuracy and the risks that students may be overly dependent on ChatGPT. They further suggested solutions to systemizing classroom dynamics between teachers and students. The study underscores the need for more research on the roles of generative AI in actual classroom settings and provides insights for future AI-integrated science learning.
{"title":"Using ChatGPT for Science Learning: A Study on Pre-service Teachers' Lesson Planning","authors":"Gyeong-Geon Lee;Xiaoming Zhai","doi":"10.1109/TLT.2024.3401457","DOIUrl":"10.1109/TLT.2024.3401457","url":null,"abstract":"While ongoing efforts have continuously emphasized the integration of ChatGPT with science teaching and learning, there are limited empirical studies exploring its actual utility in the classroom. This study aims to fill this gap by analyzing the lesson plans developed by 29 pre-service elementary teachers and assessing how they integrated ChatGPT into science learning activities. We first examined how ChatGPT was integrated with the subject domains, teaching methods/strategies, and then evaluated the lesson plans using a generative artificial intelligence (AI)-technological pedagogical and content knowledge (TPACK)-based rubric. We further examined pre-service teachers' perceptions and concerns about integrating ChatGPT into science learning. Results show a diverse number of ChatGPT applications in different science domains—e.g., Biology (9/29), Chemistry (7/29), and Earth Science (7/29). A total of 14 types of teaching methods/strategies were identified in the lesson plans. On average, the pre-service teachers' lesson plans scored high on the modified TPACK-based rubric (M = 3.29; SD = 0.91; on a 1–4 scale), indicating a reasonable envisage of integrating ChatGPT into science learning, particularly in “instructional strategies and ChatGPT” (M = 3.48; SD = 0.99). However, they scored relatively lower on exploiting ChatGPT's functions toward its full potential (M = 3.00; SD = 0.93), compared to other aspects. We also identified several inappropriate use cases of ChatGPT in lesson planning (e.g., as a source of hallucinated Internet material and technically unsupported visual guidance). Pre-service teachers anticipated ChatGPT to afford high-quality questioning, self-directed learning, individualized learning support, and formative assessment. Meanwhile, they also expressed concerns about its accuracy and the risks that students may be overly dependent on ChatGPT. They further suggested solutions to systemizing classroom dynamics between teachers and students. The study underscores the need for more research on the roles of generative AI in actual classroom settings and provides insights for future AI-integrated science learning.","PeriodicalId":49191,"journal":{"name":"IEEE Transactions on Learning Technologies","volume":"17 ","pages":"1683-1700"},"PeriodicalIF":3.7,"publicationDate":"2024-03-15","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141063211","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"教育学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-03-06DOI: 10.1109/TLT.2024.3396873
Yishen Song;Qianta Zhu;Huaibo Wang;Qinhua Zheng
Manually scoring and revising student essays has long been a time-consuming task for educators. With the rise of natural language processing techniques, automated essay scoring (AES) and automated essay revising (AER) have emerged to alleviate this burden. However, current AES and AER models require large amounts of training data and lack generalizability, which makes them hard to implement in daily teaching activities. Moreover, online sites offering AES and AER services charge high fees and have security issues uploading student content. In light of these challenges and recognizing the advancements in large language models (LLMs), we aim to fill these research gaps by analyzing the performance of open-source LLMs when accomplishing AES and AER tasks. Using a human-scored essay dataset ( n = 600) collected in an online assessment, we implemented zero-shot, few-shot, and p-tuning AES methods based on the LLMs and conducted a human–machine consistency check. We conducted a similarity test and a score difference test for the results of AER with LLMs support. The human–machine consistency check result shows that the performance of open-source LLMs with a 10 B parameter size in the AES task is close to that of some deep-learning baseline models, and it can be improved by integrating the comment with the score into the shot or training continuous prompts. The similarity test and score difference test results show that open-source LLMs can effectively accomplish the AER task, improving the quality of the essays while ensuring that the revision results are similar to the original essays. This study reveals a practical path to cost-effectively, time-efficiently, and content-safely assisting teachers with student essay scoring and revising using open-source LLMs.
{"title":"Automated Essay Scoring and Revising Based on Open-Source Large Language Models","authors":"Yishen Song;Qianta Zhu;Huaibo Wang;Qinhua Zheng","doi":"10.1109/TLT.2024.3396873","DOIUrl":"10.1109/TLT.2024.3396873","url":null,"abstract":"Manually scoring and revising student essays has long been a time-consuming task for educators. With the rise of natural language processing techniques, automated essay scoring (AES) and automated essay revising (AER) have emerged to alleviate this burden. However, current AES and AER models require large amounts of training data and lack generalizability, which makes them hard to implement in daily teaching activities. Moreover, online sites offering AES and AER services charge high fees and have security issues uploading student content. In light of these challenges and recognizing the advancements in large language models (LLMs), we aim to fill these research gaps by analyzing the performance of open-source LLMs when accomplishing AES and AER tasks. Using a human-scored essay dataset (\u0000<italic>n</i>\u0000 = 600) collected in an online assessment, we implemented zero-shot, few-shot, and p-tuning AES methods based on the LLMs and conducted a human–machine consistency check. We conducted a similarity test and a score difference test for the results of AER with LLMs support. The human–machine consistency check result shows that the performance of open-source LLMs with a 10 B parameter size in the AES task is close to that of some deep-learning baseline models, and it can be improved by integrating the comment with the score into the shot or training continuous prompts. The similarity test and score difference test results show that open-source LLMs can effectively accomplish the AER task, improving the quality of the essays while ensuring that the revision results are similar to the original essays. This study reveals a practical path to cost-effectively, time-efficiently, and content-safely assisting teachers with student essay scoring and revising using open-source LLMs.","PeriodicalId":49191,"journal":{"name":"IEEE Transactions on Learning Technologies","volume":"17 ","pages":"1920-1930"},"PeriodicalIF":2.9,"publicationDate":"2024-03-06","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"140885774","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"教育学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-03-05DOI: 10.1109/TLT.2024.3372894
Israel Ulises Cayetano-Jiménez;Erick Axel Martinez-Ríos;Rogelio Bustamante-Bello;Ricardo A. Ramírez-Mendoza;María Soledad Ramírez-Montoya
Educational robotics (ER) is a discipline of applied robotics focused on teaching robot design, analysis, application, and operation. Traditionally, ER has favored rigid robots, overlooking the potential of soft robots (SRs). While rigid robots offer insights into dynamics, kinematics, and control, they have limitations in exploring the depths of mechanical design and material properties. In this regard, SRs present an opportunity to expand educational topics and activities in robotics through their unique bioinspired properties and accessibility. Despite their promise, there is a notable lack of research on SRs as educational tools, limiting the identification of research avenues that could promote their adoption in educational settings. This study conducts a systematic literature review to elucidate the impact of SRs across academic levels, pedagogical strategies, prevalent artificial muscles, educational activities, and assessment methods. The findings indicate a significant focus on K-12 workshops utilizing soft pneumatic actuators. Furthermore, SRs have fostered the development of fabrication and mechanical design skills beyond mere programming tasks. However, there is a shortage of studies analyzing their use in higher education or their impact on learning outcomes, suggesting a critical need for comprehensive evaluations to determine their effectiveness, rather than solely relying on surveys for student feedback. Thus, there is an opportunity to explore and evaluate the use of SRs in more advanced settings and multidisciplinary activities, urging for rigorous assessments of their influence on learning outcomes. By undertaking this, we aim to provide a foundation for integrating SRs into the ER curriculum, potentially transforming teaching methodologies and enriching students' learning experiences.
{"title":"Experimenting With Soft Robotics in Education: A Systematic Literature Review From 2006 to 2022","authors":"Israel Ulises Cayetano-Jiménez;Erick Axel Martinez-Ríos;Rogelio Bustamante-Bello;Ricardo A. Ramírez-Mendoza;María Soledad Ramírez-Montoya","doi":"10.1109/TLT.2024.3372894","DOIUrl":"10.1109/TLT.2024.3372894","url":null,"abstract":"Educational robotics (ER) is a discipline of applied robotics focused on teaching robot design, analysis, application, and operation. Traditionally, ER has favored rigid robots, overlooking the potential of soft robots (SRs). While rigid robots offer insights into dynamics, kinematics, and control, they have limitations in exploring the depths of mechanical design and material properties. In this regard, SRs present an opportunity to expand educational topics and activities in robotics through their unique bioinspired properties and accessibility. Despite their promise, there is a notable lack of research on SRs as educational tools, limiting the identification of research avenues that could promote their adoption in educational settings. This study conducts a systematic literature review to elucidate the impact of SRs across academic levels, pedagogical strategies, prevalent artificial muscles, educational activities, and assessment methods. The findings indicate a significant focus on K-12 workshops utilizing soft pneumatic actuators. Furthermore, SRs have fostered the development of fabrication and mechanical design skills beyond mere programming tasks. However, there is a shortage of studies analyzing their use in higher education or their impact on learning outcomes, suggesting a critical need for comprehensive evaluations to determine their effectiveness, rather than solely relying on surveys for student feedback. Thus, there is an opportunity to explore and evaluate the use of SRs in more advanced settings and multidisciplinary activities, urging for rigorous assessments of their influence on learning outcomes. By undertaking this, we aim to provide a foundation for integrating SRs into the ER curriculum, potentially transforming teaching methodologies and enriching students' learning experiences.","PeriodicalId":49191,"journal":{"name":"IEEE Transactions on Learning Technologies","volume":"17 ","pages":"1261-1278"},"PeriodicalIF":3.7,"publicationDate":"2024-03-05","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=10460415","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"140045730","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"教育学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-03-04DOI: 10.1109/TLT.2024.3372508
Yasar C. Kakdas;Sinan Kockara;Tansel Halic;Doga Demirel
This article presents a 3-D medical simulation that employs reinforcement learning (RL) and interactive RL (IRL) to teach and assess the procedure of donning and doffing personal protective equipment (PPE). The simulation is motivated by the need for effective, safe, and remote training techniques in medicine, particularly in light of the COVID-19 pandemic. The simulation has two modes: a tutorial mode and an assessment mode. In the tutorial mode, a computer-based, ill-trained RL agent utilizes RL to learn the correct sequence of donning the PPE by trial and error. This allows students to experience many outlier cases they might not encounter in an in-class educational model. In the assessment mode, an IRL-based method is used to evaluate how effective the participant is at correcting the mistakes performed by the RL agent. Each time the RL agent interacts with the environment and performs an action, the participants provide positive or negative feedback regarding the action taken. Following the assessment, participants receive a score based on the accuracy of their feedback and the time taken for the RL agent to learn the correct sequence. An experiment was conducted using two groups, each consisting of ten participants. The first group received RL-assisted training for donning PPE, followed by an IRL-based assessment. Meanwhile, the second group observed a video featuring the RL agent demonstrating only the correct donning order without outlier cases, replicating traditional training, before undergoing the same assessment as the first group. Results showed that RL-assisted training with many outlier cases was more effective than traditional training with only regular cases. Moreover, combining RL with IRL significantly enhanced the participants' performance. Notably, 90% of the participants finished the assessment with perfect scores within three iterations. In contrast, only 10% of those who did not engage in RL-assisted training finished the assessment with a perfect score, highlighting the substantial impact of RL and IRL integration on participants’ overall achievement.
{"title":"Enhancing Medical Training Through Learning From Mistakes by Interacting With an Ill-Trained Reinforcement Learning Agent","authors":"Yasar C. Kakdas;Sinan Kockara;Tansel Halic;Doga Demirel","doi":"10.1109/TLT.2024.3372508","DOIUrl":"10.1109/TLT.2024.3372508","url":null,"abstract":"This article presents a 3-D medical simulation that employs reinforcement learning (RL) and interactive RL (IRL) to teach and assess the procedure of donning and doffing personal protective equipment (PPE). The simulation is motivated by the need for effective, safe, and remote training techniques in medicine, particularly in light of the COVID-19 pandemic. The simulation has two modes: a tutorial mode and an assessment mode. In the tutorial mode, a computer-based, ill-trained RL agent utilizes RL to learn the correct sequence of donning the PPE by trial and error. This allows students to experience many outlier cases they might not encounter in an in-class educational model. In the assessment mode, an IRL-based method is used to evaluate how effective the participant is at correcting the mistakes performed by the RL agent. Each time the RL agent interacts with the environment and performs an action, the participants provide positive or negative feedback regarding the action taken. Following the assessment, participants receive a score based on the accuracy of their feedback and the time taken for the RL agent to learn the correct sequence. An experiment was conducted using two groups, each consisting of ten participants. The first group received RL-assisted training for donning PPE, followed by an IRL-based assessment. Meanwhile, the second group observed a video featuring the RL agent demonstrating only the correct donning order without outlier cases, replicating traditional training, before undergoing the same assessment as the first group. Results showed that RL-assisted training with many outlier cases was more effective than traditional training with only regular cases. Moreover, combining RL with IRL significantly enhanced the participants' performance. Notably, 90% of the participants finished the assessment with perfect scores within three iterations. In contrast, only 10% of those who did not engage in RL-assisted training finished the assessment with a perfect score, highlighting the substantial impact of RL and IRL integration on participants’ overall achievement.","PeriodicalId":49191,"journal":{"name":"IEEE Transactions on Learning Technologies","volume":"17 ","pages":"1248-1260"},"PeriodicalIF":3.7,"publicationDate":"2024-03-04","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"140037300","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"教育学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
京公网安备 11010802042870号
京ICP备2023020795号-1
扫码关注我们
求助内容：
应助结果提醒方式：