Harley Harwood , Nicolas Roulin , Muhammad Zafar Iqbal
{"title":"\"你能做到的,我都能做到\":考察 ChatGPT 在专业课程入学情景判断测试中的应用","authors":"Harley Harwood , Nicolas Roulin , Muhammad Zafar Iqbal","doi":"10.1016/j.jvb.2024.104013","DOIUrl":null,"url":null,"abstract":"<div><p>We explored the transformative impact of ChatGPT on applicants' responses and performance in situational judgement tests (SJTs), as well as the role played by faking-prevention mechanisms, in two complementary studies. Study 1 examined how the availability of ChatGPT influenced response content and performance of real applicants (<em>N</em> = 107,805), who completed an SJT for admission before vs. after the release of the technology. We found only small differences in content (e.g., slightly less “authentic” words used) and performance (slight score improvements when controlling for response length, no differences otherwise). In Study 2, we used an experimental approach with (<em>N</em> = 138) Prolific participants completing a mock SJT, while being instructed to use ChatGPT when responding (vs. use online resources or no resources). We found only slightly higher SJT scores for the ChatGPT users, but no difference in response content. Additionally, GPTZero (i.e., a popular AI detection tool) struggled to detect ChatGPT content, and generated many false positives, in both studies. This research advances our understanding of how the release and popularization of ChatGPT can influence applicant behaviors. Given the “arms race” nature of applicant selection, they also highlight the importance of designing assessments to prevent or limit faking. Yet, the ever-evolving nature of AI calls for continuous research on the topic.</p></div>","PeriodicalId":51344,"journal":{"name":"Journal of Vocational Behavior","volume":"154 ","pages":"Article 104013"},"PeriodicalIF":5.2000,"publicationDate":"2024-06-24","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.sciencedirect.com/science/article/pii/S000187912400054X/pdfft?md5=6a082cfa624b673fef798ba580a09e03&pid=1-s2.0-S000187912400054X-main.pdf","citationCount":"0","resultStr":"{\"title\":\"“Anything you can do, I can do”: Examining the use of ChatGPT in situational judgement tests for professional program admission\",\"authors\":\"Harley Harwood , Nicolas Roulin , Muhammad Zafar Iqbal\",\"doi\":\"10.1016/j.jvb.2024.104013\",\"DOIUrl\":null,\"url\":null,\"abstract\":\"<div><p>We explored the transformative impact of ChatGPT on applicants' responses and performance in situational judgement tests (SJTs), as well as the role played by faking-prevention mechanisms, in two complementary studies. Study 1 examined how the availability of ChatGPT influenced response content and performance of real applicants (<em>N</em> = 107,805), who completed an SJT for admission before vs. after the release of the technology. We found only small differences in content (e.g., slightly less “authentic” words used) and performance (slight score improvements when controlling for response length, no differences otherwise). In Study 2, we used an experimental approach with (<em>N</em> = 138) Prolific participants completing a mock SJT, while being instructed to use ChatGPT when responding (vs. use online resources or no resources). We found only slightly higher SJT scores for the ChatGPT users, but no difference in response content. Additionally, GPTZero (i.e., a popular AI detection tool) struggled to detect ChatGPT content, and generated many false positives, in both studies. This research advances our understanding of how the release and popularization of ChatGPT can influence applicant behaviors. Given the “arms race” nature of applicant selection, they also highlight the importance of designing assessments to prevent or limit faking. Yet, the ever-evolving nature of AI calls for continuous research on the topic.</p></div>\",\"PeriodicalId\":51344,\"journal\":{\"name\":\"Journal of Vocational Behavior\",\"volume\":\"154 \",\"pages\":\"Article 104013\"},\"PeriodicalIF\":5.2000,\"publicationDate\":\"2024-06-24\",\"publicationTypes\":\"Journal Article\",\"fieldsOfStudy\":null,\"isOpenAccess\":false,\"openAccessPdf\":\"https://www.sciencedirect.com/science/article/pii/S000187912400054X/pdfft?md5=6a082cfa624b673fef798ba580a09e03&pid=1-s2.0-S000187912400054X-main.pdf\",\"citationCount\":\"0\",\"resultStr\":null,\"platform\":\"Semanticscholar\",\"paperid\":null,\"PeriodicalName\":\"Journal of Vocational Behavior\",\"FirstCategoryId\":\"102\",\"ListUrlMain\":\"https://www.sciencedirect.com/science/article/pii/S000187912400054X\",\"RegionNum\":1,\"RegionCategory\":\"心理学\",\"ArticlePicture\":[],\"TitleCN\":null,\"AbstractTextCN\":null,\"PMCID\":null,\"EPubDate\":\"\",\"PubModel\":\"\",\"JCR\":\"Q1\",\"JCRName\":\"PSYCHOLOGY, APPLIED\",\"Score\":null,\"Total\":0}","platform":"Semanticscholar","paperid":null,"PeriodicalName":"Journal of Vocational Behavior","FirstCategoryId":"102","ListUrlMain":"https://www.sciencedirect.com/science/article/pii/S000187912400054X","RegionNum":1,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"PSYCHOLOGY, APPLIED","Score":null,"Total":0}
“Anything you can do, I can do”: Examining the use of ChatGPT in situational judgement tests for professional program admission
We explored the transformative impact of ChatGPT on applicants' responses and performance in situational judgement tests (SJTs), as well as the role played by faking-prevention mechanisms, in two complementary studies. Study 1 examined how the availability of ChatGPT influenced response content and performance of real applicants (N = 107,805), who completed an SJT for admission before vs. after the release of the technology. We found only small differences in content (e.g., slightly less “authentic” words used) and performance (slight score improvements when controlling for response length, no differences otherwise). In Study 2, we used an experimental approach with (N = 138) Prolific participants completing a mock SJT, while being instructed to use ChatGPT when responding (vs. use online resources or no resources). We found only slightly higher SJT scores for the ChatGPT users, but no difference in response content. Additionally, GPTZero (i.e., a popular AI detection tool) struggled to detect ChatGPT content, and generated many false positives, in both studies. This research advances our understanding of how the release and popularization of ChatGPT can influence applicant behaviors. Given the “arms race” nature of applicant selection, they also highlight the importance of designing assessments to prevent or limit faking. Yet, the ever-evolving nature of AI calls for continuous research on the topic.
期刊介绍:
The Journal of Vocational Behavior publishes original empirical and theoretical articles offering unique insights into the realms of career choice, career development, and work adjustment across the lifespan. These contributions are not only valuable for academic exploration but also find applications in counseling and career development programs across diverse sectors such as colleges, universities, business, industry, government, and the military.
The primary focus of the journal centers on individual decision-making regarding work and careers, prioritizing investigations into personal career choices rather than organizational or employer-level variables. Example topics encompass a broad range, from initial career choices (e.g., choice of major, initial work or organization selection, organizational attraction) to the development of a career, work transitions, work-family management, and attitudes within the workplace (such as work commitment, multiple role management, and turnover).