{"title":"Using Artificial Intelligence (AI) As An External Examiner","authors":"Tayyaba Azhar, Kinza Aslam, Zakia Saleem, Ahsan Sethi, Tahseen Fatima","doi":"10.51273/esc23.251319323","DOIUrl":null,"url":null,"abstract":"Objective: To access the validity of ChatGPT on AI assisted tool for evaluating essay questions. Material and Methods: This was a cross-sectional quantitative study conducted at University College of Medicine and Dentistry from June till August 2023. Eighteen questions were selected from fifteen exit tests of Certificate in HPE course. Each of the answers were independently graded by two assessors with doctorate in HPE. The same answers were then reevaluated using ChatGPT. The inter-rater reliability was determined using Kappa test. Results: The agreement between ChatGPT and examiner scores varied on various items. Weak agreement was observed for questions 8 and 9, moderate agreement for questions 2, 3, and 5, and strong kappa agreement for questions 1, 4, 6, and 7. Conclusion: Artificial intelligence assisted tools such as ChatGPT is a reality but its use in assessing essay questions would require massive training data from expert assessors. Once appropriately trained, it may replicate assessment decisions across the full range of subject.","PeriodicalId":11923,"journal":{"name":"Esculapio","volume":"15 1","pages":""},"PeriodicalIF":0.0000,"publicationDate":"2023-11-08","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Esculapio","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/10.51273/esc23.251319323","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"","JCRName":"","Score":null,"Total":0}
引用次数: 0
Abstract
Objective: To access the validity of ChatGPT on AI assisted tool for evaluating essay questions. Material and Methods: This was a cross-sectional quantitative study conducted at University College of Medicine and Dentistry from June till August 2023. Eighteen questions were selected from fifteen exit tests of Certificate in HPE course. Each of the answers were independently graded by two assessors with doctorate in HPE. The same answers were then reevaluated using ChatGPT. The inter-rater reliability was determined using Kappa test. Results: The agreement between ChatGPT and examiner scores varied on various items. Weak agreement was observed for questions 8 and 9, moderate agreement for questions 2, 3, and 5, and strong kappa agreement for questions 1, 4, 6, and 7. Conclusion: Artificial intelligence assisted tools such as ChatGPT is a reality but its use in assessing essay questions would require massive training data from expert assessors. Once appropriately trained, it may replicate assessment decisions across the full range of subject.
京公网安备 11010802042870号
京ICP备2023020795号-1
分享
求助内容：
应助结果提醒方式：
扫码关注我们
