Integrating peer assessment cycle into ChatGPT for STEM education: A randomised controlled trial on knowledge, skills, and attitudes enhancement

IF 5.1 2区教育学 Q1 EDUCATION & EDUCATIONAL RESEARCH Journal of Computer Assisted Learning Pub Date : 2024-10-23 DOI:10.1111/jcal.13085

Ting-Ting Wu, Hsin-Yu Lee, Pei-Hua Chen, Chia-Ju Lin, Yueh-Min Huang

{"title":"Integrating peer assessment cycle into ChatGPT for STEM education: A randomised controlled trial on knowledge, skills, and attitudes enhancement","authors":"Ting-Ting Wu, Hsin-Yu Lee, Pei-Hua Chen, Chia-Ju Lin, Yueh-Min Huang","doi":"10.1111/jcal.13085","DOIUrl":null,"url":null,"abstract":"<div>\n \n \n <section>\n \n <h3> Background</h3>\n \n <p>Science, Technology, Engineering, and Mathematics (STEM) education in Asian universities struggles to integrate Knowledge, Skills, and Attitudes (KSA) due to large classes and student reluctance. While ChatGPT offers solutions, its conventional use may hinder independent critical thinking.</p>\n </section>\n \n <section>\n \n <h3> Objectives</h3>\n \n <p>This study introduces PA-GPT, using ChatGPT as a “virtual peer” in peer assessments to promote active learning and enhance knowledge, higher-order thinking skills (HOTS), and attitudes—the core of KSA in STEM.</p>\n </section>\n \n <section>\n \n <h3> Methods</h3>\n \n <p>A randomised controlled trial involved 61 first-year engineering students (43 males, 18 females) from a university in Southern Taiwan enrolled in “Network Embedded Systems and Applications.” Participants, all with prior ChatGPT experience but no programming background, were purposively sampled. They were randomly assigned to the experimental group (<i>n</i> = 31) using PA-GPT or the control group (<i>n</i> = 30) using traditional ChatGPT. Over 8 weeks, data were collected using pre- and post-tests: a knowledge construction test (20 items, <i>α</i> = 0.85); a HOTS scale (<i>α</i> = 0.78–0.83) measuring critical thinking, problem-solving, and creativity; and the S-STEM questionnaire (<i>α</i> >0.80) assessing attitudes towards STEM subjects and 21st-century learning. ANCOVA analysed the data, controlling for pre-test scores, and Levene's test checked homogeneity of variances.</p>\n </section>\n \n <section>\n \n <h3> Results and Conclusions</h3>\n \n <p>ANCOVA results showed that PA-GPT significantly outperformed traditional ChatGPT in enhancing knowledge construction (<i>F</i> = 9.89, <i>p</i> = 0.002), critical thinking (<i>F</i> = 37.00, <i>p</i> < 0.001), problem-solving (<i>F</i> = 9.40, <i>p</i> = 0.003), creativity (F = 7.22, <i>p</i> = 0.009), and attitudes towards mathematics (<i>F</i> = 25.52, <i>p</i> < 0.001), engineering/technology (<i>F</i> = 16.06, <i>p</i> < 0.001), and 21st-century learning (<i>F</i> = 26.38, <i>p</i> < 0.001). These findings demonstrate that PA-GPT effectively addresses challenges in student engagement and HOTS development in STEM education by simulating peer interactions. Peer Assessment with ChatGPT (PA-GPT) promotes active learning and self-reflection, potentially revolutionising AI-assisted education in large class settings. This study provides pioneering evidence for the effectiveness of AI-driven peer assessment in enhancing comprehensive STEM competencies, offering a promising direction for future educational technology integration.</p>\n </section>\n </div>","PeriodicalId":48071,"journal":{"name":"Journal of Computer Assisted Learning","volume":"41 1","pages":""},"PeriodicalIF":5.1000,"publicationDate":"2024-10-23","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Journal of Computer Assisted Learning","FirstCategoryId":"95","ListUrlMain":"https://onlinelibrary.wiley.com/doi/10.1111/jcal.13085","RegionNum":2,"RegionCategory":"教育学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"EDUCATION & EDUCATIONAL RESEARCH","Score":null,"Total":0}

引用次数: 0

Abstract

Background

Science, Technology, Engineering, and Mathematics (STEM) education in Asian universities struggles to integrate Knowledge, Skills, and Attitudes (KSA) due to large classes and student reluctance. While ChatGPT offers solutions, its conventional use may hinder independent critical thinking.

Objectives

This study introduces PA-GPT, using ChatGPT as a “virtual peer” in peer assessments to promote active learning and enhance knowledge, higher-order thinking skills (HOTS), and attitudes—the core of KSA in STEM.

Methods

A randomised controlled trial involved 61 first-year engineering students (43 males, 18 females) from a university in Southern Taiwan enrolled in “Network Embedded Systems and Applications.” Participants, all with prior ChatGPT experience but no programming background, were purposively sampled. They were randomly assigned to the experimental group (n = 31) using PA-GPT or the control group (n = 30) using traditional ChatGPT. Over 8 weeks, data were collected using pre- and post-tests: a knowledge construction test (20 items, α = 0.85); a HOTS scale (α = 0.78–0.83) measuring critical thinking, problem-solving, and creativity; and the S-STEM questionnaire (α >0.80) assessing attitudes towards STEM subjects and 21st-century learning. ANCOVA analysed the data, controlling for pre-test scores, and Levene's test checked homogeneity of variances.

Results and Conclusions

ANCOVA results showed that PA-GPT significantly outperformed traditional ChatGPT in enhancing knowledge construction (F = 9.89, p = 0.002), critical thinking (F = 37.00, p < 0.001), problem-solving (F = 9.40, p = 0.003), creativity (F = 7.22, p = 0.009), and attitudes towards mathematics (F = 25.52, p < 0.001), engineering/technology (F = 16.06, p < 0.001), and 21st-century learning (F = 26.38, p < 0.001). These findings demonstrate that PA-GPT effectively addresses challenges in student engagement and HOTS development in STEM education by simulating peer interactions. Peer Assessment with ChatGPT (PA-GPT) promotes active learning and self-reflection, potentially revolutionising AI-assisted education in large class settings. This study provides pioneering evidence for the effectiveness of AI-driven peer assessment in enhancing comprehensive STEM competencies, offering a promising direction for future educational technology integration.

查看原文

微信好友朋友圈 QQ好友复制链接

本刊更多论文

将同行评估周期融入STEM教育ChatGPT：一项关于知识、技能和态度增强的随机对照试验

亚洲大学的科学、技术、工程和数学（STEM）教育由于大班和学生的不情愿而难以整合知识、技能和态度（KSA）。虽然ChatGPT提供了解决方案，但它的常规使用可能会阻碍独立的批判性思维。本研究引入PA-GPT，使用ChatGPT作为同伴评估中的“虚拟同伴”，以促进主动学习，提高STEM中KSA的核心知识、高阶思维技能（HOTS）和态度。方法采用随机对照试验方法，选取台湾南部一所大学“网络嵌入式系统与应用”专业的61名一年级工科学生（男43名，女18名）。参与者之前都有ChatGPT的经验，但没有编程背景，有意取样。随机分为使用PA-GPT的实验组（n = 31）和使用传统ChatGPT的对照组（n = 30）。在8周内，采用前测和后测收集数据：知识构建测试（20项，α = 0.85）；HOTS量表（α = 0.78-0.83）衡量批判性思维、解决问题和创造力；S-STEM问卷（α >0.80）评估对STEM科目和21世纪学习的态度。ANCOVA分析了数据，控制了测试前的分数，Levene的测试检查了方差的同质性。ANCOVA结果显示，PA-GPT在知识构建（F = 9.89, p = 0.002）、批判性思维（F = 37.00, p < 0.001）、问题解决（F = 9.40, p = 0.003）、创造力（F = 7.22, p = 0.009）、对数学（F = 25.52, p < 0.001）、工程/技术（F = 16.06, p < 0.001）和21世纪学习（F = 26.38, p < 0.001）的态度方面显著优于传统ChatGPT。这些发现表明，PA-GPT通过模拟同伴互动，有效地解决了STEM教育中学生参与和HOTS发展的挑战。基于ChatGPT的同伴评估（PA-GPT）促进了主动学习和自我反思，可能会在大班环境中彻底改变人工智能辅助教育。本研究为人工智能驱动的同行评估在提高STEM综合能力方面的有效性提供了开创性的证据，为未来的教育技术整合提供了一个有希望的方向。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文去求助

来源期刊

Journal of Computer Assisted Learning EDUCATION & EDUCATIONAL RESEARCH-

CiteScore

9.70

自引率

6.00%

发文量

116

期刊介绍： The Journal of Computer Assisted Learning is an international peer-reviewed journal which covers the whole range of uses of information and communication technology to support learning and knowledge exchange. It aims to provide a medium for communication among researchers as well as a channel linking researchers, practitioners, and policy makers. JCAL is also a rich source of material for master and PhD students in areas such as educational psychology, the learning sciences, instructional technology, instructional design, collaborative learning, intelligent learning systems, learning analytics, open, distance and networked learning, and educational evaluation and assessment. This is the case for formal (e.g., schools), non-formal (e.g., workplace learning) and informal learning (e.g., museums and libraries) situations and environments. Volumes often include one Special Issue which these provides readers with a broad and in-depth perspective on a specific topic. First published in 1985, JCAL continues to have the aim of making the outcomes of contemporary research and experience accessible. During this period there have been major technological advances offering new opportunities and approaches in the use of a wide range of technologies to support learning and knowledge transfer more generally. There is currently much emphasis on the use of network functionality and the challenges its appropriate uses pose to teachers/tutors working with students locally and at a distance. JCAL welcomes: -Empirical reports, single studies or programmatic series of studies on the use of computers and information technologies in learning and assessment -Critical and original meta-reviews of literature on the use of computers for learning -Empirical studies on the design and development of innovative technology-based systems for learning -Conceptual articles on issues relating to the Aims and Scope