Music teachers’ labeling accuracy and quality ratings of lesson plans by artificial intelligence (AI) and humans

IF 1.3 3区教育学 Q2 EDUCATION & EDUCATIONAL RESEARCH International Journal of Music Education Pub Date : 2024-05-08 DOI:10.1177/02557614241249163

Patrick K Cooper

{"title":"Music teachers’ labeling accuracy and quality ratings of lesson plans by artificial intelligence (AI) and humans","authors":"Patrick K Cooper","doi":"10.1177/02557614241249163","DOIUrl":null,"url":null,"abstract":"This study explored the potential of artificial intelligence (ChatGPT) to generate lesson plans for music classes that were indistinguishable from music lesson plans created by humans, with current music teachers as assessors. Fifty-six assessors made a total of 410 ratings across eight lesson plans, assigning a quality score to each lesson plan and labeling if they believed each lesson plan was created by a human or generated by AI. Despite the human-made lesson plans being rated higher in quality as a group ( p < .01, d = 0.44), assessors were unable to accurately label if a lesson plan was created by a human or generated by AI (55% accurate overall). Labeling accuracy was positively predicted by quality scores on human-made lesson plans and previous personal use of AI, while accuracy was negatively predicted by quality scores on AI-generated lesson plans and perception of how useful AI will be in the future. Open-ended responses from 42 teachers suggested assessors used three factors when making evaluations: specific details, evidence of classroom knowledge, and wording. Implications provide suggestions for how music teachers can use prompt engineering with a GPT model to create a virtual assistant or Intelligent Tutor System (ITS) for their classroom.","PeriodicalId":46623,"journal":{"name":"International Journal of Music Education","volume":"50 1","pages":""},"PeriodicalIF":1.3000,"publicationDate":"2024-05-08","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"International Journal of Music Education","FirstCategoryId":"95","ListUrlMain":"https://doi.org/10.1177/02557614241249163","RegionNum":3,"RegionCategory":"教育学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q2","JCRName":"EDUCATION & EDUCATIONAL RESEARCH","Score":null,"Total":0}

引用次数: 0

Abstract

This study explored the potential of artificial intelligence (ChatGPT) to generate lesson plans for music classes that were indistinguishable from music lesson plans created by humans, with current music teachers as assessors. Fifty-six assessors made a total of 410 ratings across eight lesson plans, assigning a quality score to each lesson plan and labeling if they believed each lesson plan was created by a human or generated by AI. Despite the human-made lesson plans being rated higher in quality as a group ( p < .01, d = 0.44), assessors were unable to accurately label if a lesson plan was created by a human or generated by AI (55% accurate overall). Labeling accuracy was positively predicted by quality scores on human-made lesson plans and previous personal use of AI, while accuracy was negatively predicted by quality scores on AI-generated lesson plans and perception of how useful AI will be in the future. Open-ended responses from 42 teachers suggested assessors used three factors when making evaluations: specific details, evidence of classroom knowledge, and wording. Implications provide suggestions for how music teachers can use prompt engineering with a GPT model to create a virtual assistant or Intelligent Tutor System (ITS) for their classroom.

查看原文

微信好友朋友圈 QQ好友复制链接

本刊更多论文

音乐教师对人工智能（AI）和人类教案的标注准确性和质量评级

本研究探索了人工智能（ChatGPT）为音乐课生成教案的潜力，这些教案与人类创建的音乐教案无异，由现任音乐教师担任评估员。56 位评估者对 8 份教案共进行了 410 次评分，给每份教案打了质量分，并标注了他们认为每份教案是由人类制作的还是由人工智能生成的。尽管人类制作的教案作为一个群体被评为质量较高（p < .01，d = 0.44），但评估者无法准确标出教案是由人类制作的还是由人工智能生成的（总体准确率为 55%）。人工智能教案的质量得分和个人以前对人工智能的使用情况对标注准确性有正面影响，而人工智能生成的教案的质量得分和对人工智能未来有用性的看法对标注准确性有负面影响。42 位教师的开放式回答表明，评估者在进行评估时使用了三个因素：具体细节、课堂知识证据和措辞。研究启示为音乐教师如何利用提示工程和 GPT 模型为课堂创建虚拟助手或智能辅导系统（ITS）提供了建议。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文去求助

来源期刊

International Journal of Music Education Multiple-

CiteScore

3.20

自引率

11.10%

发文量

期刊介绍： The International Journal of Music Education (IJME) is a peer-reviewed journal published by the International Society for Music Education (ISME) four times a year. Manuscripts published are scholarly works, representing empirical research in a variety of modalities. They enhance knowledge regarding the teaching and learning of music with a special interest toward an international constituency. Manuscripts report results of quantitative or qualitative research studies, summarize bodies or research, present theories, models, or philosophical positions, etc. Papers show relevance to advancing the practice of music teaching and learning at all age levels with issues of direct concern to the classroom or studio, in school and out, private and group instruction. All manuscripts should contain evidence of a scholarly approach and be situated within the current literature. Implications for learning and teaching of music should be clearly stated, relevant, contemporary, and of interest to an international readership.