"你能做到的,我都能做到":考察 ChatGPT 在专业课程入学情景判断测试中的应用

IF 5.2 1区 心理学 Q1 PSYCHOLOGY, APPLIED Journal of Vocational Behavior Pub Date : 2024-06-24 DOI:10.1016/j.jvb.2024.104013
Harley Harwood , Nicolas Roulin , Muhammad Zafar Iqbal
{"title":"\"你能做到的,我都能做到\":考察 ChatGPT 在专业课程入学情景判断测试中的应用","authors":"Harley Harwood ,&nbsp;Nicolas Roulin ,&nbsp;Muhammad Zafar Iqbal","doi":"10.1016/j.jvb.2024.104013","DOIUrl":null,"url":null,"abstract":"<div><p>We explored the transformative impact of ChatGPT on applicants' responses and performance in situational judgement tests (SJTs), as well as the role played by faking-prevention mechanisms, in two complementary studies. Study 1 examined how the availability of ChatGPT influenced response content and performance of real applicants (<em>N</em> = 107,805), who completed an SJT for admission before vs. after the release of the technology. We found only small differences in content (e.g., slightly less “authentic” words used) and performance (slight score improvements when controlling for response length, no differences otherwise). In Study 2, we used an experimental approach with (<em>N</em> = 138) Prolific participants completing a mock SJT, while being instructed to use ChatGPT when responding (vs. use online resources or no resources). We found only slightly higher SJT scores for the ChatGPT users, but no difference in response content. Additionally, GPTZero (i.e., a popular AI detection tool) struggled to detect ChatGPT content, and generated many false positives, in both studies. This research advances our understanding of how the release and popularization of ChatGPT can influence applicant behaviors. Given the “arms race” nature of applicant selection, they also highlight the importance of designing assessments to prevent or limit faking. Yet, the ever-evolving nature of AI calls for continuous research on the topic.</p></div>","PeriodicalId":51344,"journal":{"name":"Journal of Vocational Behavior","volume":"154 ","pages":"Article 104013"},"PeriodicalIF":5.2000,"publicationDate":"2024-06-24","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.sciencedirect.com/science/article/pii/S000187912400054X/pdfft?md5=6a082cfa624b673fef798ba580a09e03&pid=1-s2.0-S000187912400054X-main.pdf","citationCount":"0","resultStr":"{\"title\":\"“Anything you can do, I can do”: Examining the use of ChatGPT in situational judgement tests for professional program admission\",\"authors\":\"Harley Harwood ,&nbsp;Nicolas Roulin ,&nbsp;Muhammad Zafar Iqbal\",\"doi\":\"10.1016/j.jvb.2024.104013\",\"DOIUrl\":null,\"url\":null,\"abstract\":\"<div><p>We explored the transformative impact of ChatGPT on applicants' responses and performance in situational judgement tests (SJTs), as well as the role played by faking-prevention mechanisms, in two complementary studies. Study 1 examined how the availability of ChatGPT influenced response content and performance of real applicants (<em>N</em> = 107,805), who completed an SJT for admission before vs. after the release of the technology. We found only small differences in content (e.g., slightly less “authentic” words used) and performance (slight score improvements when controlling for response length, no differences otherwise). In Study 2, we used an experimental approach with (<em>N</em> = 138) Prolific participants completing a mock SJT, while being instructed to use ChatGPT when responding (vs. use online resources or no resources). We found only slightly higher SJT scores for the ChatGPT users, but no difference in response content. Additionally, GPTZero (i.e., a popular AI detection tool) struggled to detect ChatGPT content, and generated many false positives, in both studies. This research advances our understanding of how the release and popularization of ChatGPT can influence applicant behaviors. Given the “arms race” nature of applicant selection, they also highlight the importance of designing assessments to prevent or limit faking. Yet, the ever-evolving nature of AI calls for continuous research on the topic.</p></div>\",\"PeriodicalId\":51344,\"journal\":{\"name\":\"Journal of Vocational Behavior\",\"volume\":\"154 \",\"pages\":\"Article 104013\"},\"PeriodicalIF\":5.2000,\"publicationDate\":\"2024-06-24\",\"publicationTypes\":\"Journal Article\",\"fieldsOfStudy\":null,\"isOpenAccess\":false,\"openAccessPdf\":\"https://www.sciencedirect.com/science/article/pii/S000187912400054X/pdfft?md5=6a082cfa624b673fef798ba580a09e03&pid=1-s2.0-S000187912400054X-main.pdf\",\"citationCount\":\"0\",\"resultStr\":null,\"platform\":\"Semanticscholar\",\"paperid\":null,\"PeriodicalName\":\"Journal of Vocational Behavior\",\"FirstCategoryId\":\"102\",\"ListUrlMain\":\"https://www.sciencedirect.com/science/article/pii/S000187912400054X\",\"RegionNum\":1,\"RegionCategory\":\"心理学\",\"ArticlePicture\":[],\"TitleCN\":null,\"AbstractTextCN\":null,\"PMCID\":null,\"EPubDate\":\"\",\"PubModel\":\"\",\"JCR\":\"Q1\",\"JCRName\":\"PSYCHOLOGY, APPLIED\",\"Score\":null,\"Total\":0}","platform":"Semanticscholar","paperid":null,"PeriodicalName":"Journal of Vocational Behavior","FirstCategoryId":"102","ListUrlMain":"https://www.sciencedirect.com/science/article/pii/S000187912400054X","RegionNum":1,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"PSYCHOLOGY, APPLIED","Score":null,"Total":0}
引用次数: 0

摘要

我们在两项互补研究中探讨了 ChatGPT 对申请人在情境判断测试(SJT)中的反应和表现所产生的变革性影响,以及防伪机制所发挥的作用。研究 1 考察了 ChatGPT 的可用性如何影响真实申请者(N=107,805)的回答内容和表现,这些申请者在该技术发布之前和之后完成了入学 SJT。我们发现,两者在内容(如使用的 "真实 "词语略少)和成绩(在控制答题长度的情况下,得分略有提高,其他方面没有差异)上只有微小差异。在研究 2 中,我们采用了一种实验方法,让 Prolific 参与者(N = 138)完成模拟 SJT,同时指导他们在作答时使用 ChatGPT(与使用在线资源或不使用资源相比)。我们发现,ChatGPT 用户的 SJT 分数略高,但在回答内容上没有差异。此外,在这两项研究中,GPTZero(即一种流行的人工智能检测工具)都很难检测到 ChatGPT 内容,并产生了许多误报。这项研究加深了我们对 ChatGPT 的发布和普及如何影响申请人行为的理解。鉴于申请人选拔的 "军备竞赛 "性质,这些研究还强调了设计评估以防止或限制造假的重要性。然而,人工智能不断发展的本质要求我们继续对这一课题进行研究。
本文章由计算机程序翻译,如有差异,请以英文原文为准。
查看原文
分享 分享
微信好友 朋友圈 QQ好友 复制链接
本刊更多论文
“Anything you can do, I can do”: Examining the use of ChatGPT in situational judgement tests for professional program admission

We explored the transformative impact of ChatGPT on applicants' responses and performance in situational judgement tests (SJTs), as well as the role played by faking-prevention mechanisms, in two complementary studies. Study 1 examined how the availability of ChatGPT influenced response content and performance of real applicants (N = 107,805), who completed an SJT for admission before vs. after the release of the technology. We found only small differences in content (e.g., slightly less “authentic” words used) and performance (slight score improvements when controlling for response length, no differences otherwise). In Study 2, we used an experimental approach with (N = 138) Prolific participants completing a mock SJT, while being instructed to use ChatGPT when responding (vs. use online resources or no resources). We found only slightly higher SJT scores for the ChatGPT users, but no difference in response content. Additionally, GPTZero (i.e., a popular AI detection tool) struggled to detect ChatGPT content, and generated many false positives, in both studies. This research advances our understanding of how the release and popularization of ChatGPT can influence applicant behaviors. Given the “arms race” nature of applicant selection, they also highlight the importance of designing assessments to prevent or limit faking. Yet, the ever-evolving nature of AI calls for continuous research on the topic.

求助全文
通过发布文献求助,成功后即可免费获取论文全文。 去求助
来源期刊
Journal of Vocational Behavior
Journal of Vocational Behavior PSYCHOLOGY, APPLIED-
CiteScore
13.10
自引率
5.40%
发文量
85
期刊介绍: The Journal of Vocational Behavior publishes original empirical and theoretical articles offering unique insights into the realms of career choice, career development, and work adjustment across the lifespan. These contributions are not only valuable for academic exploration but also find applications in counseling and career development programs across diverse sectors such as colleges, universities, business, industry, government, and the military. The primary focus of the journal centers on individual decision-making regarding work and careers, prioritizing investigations into personal career choices rather than organizational or employer-level variables. Example topics encompass a broad range, from initial career choices (e.g., choice of major, initial work or organization selection, organizational attraction) to the development of a career, work transitions, work-family management, and attitudes within the workplace (such as work commitment, multiple role management, and turnover).
期刊最新文献
Does grade point average have a long-lasting impact on career success later in life? A resource caravans' perspective from adolescence to mid-career Give it your all or hardly give? The role of mentors' beliefs about protégé advancement potential and gender in mentoring relationships Are they more proactive or less engaged? Understanding employees' career proactivity after promotion failure through an attribution lens Humble leader, successful follower: Linking leader humility with follower career outcomes via leader competence from an implicit leadership theory perspective The psychological experience of flexibility in the workplace: How psychological job control and boundary control profiles relate to the wellbeing of flexible workers
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
现在去查看 取消
×
提示
确定
0
微信
客服QQ
Book学术公众号 扫码关注我们
反馈
×
意见反馈
请填写您的意见或建议
请填写您的手机或邮箱
已复制链接
已复制链接
快去分享给好友吧!
我知道了
×
扫码分享
扫码分享
Book学术官方微信
Book学术文献互助
Book学术文献互助群
群 号:481959085
Book学术
文献互助 智能选刊 最新文献 互助须知 联系我们:info@booksci.cn
Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。
Copyright © 2023 Book学术 All rights reserved.
ghs 京公网安备 11010802042870号 京ICP备2023020795号-1