{"title":"Experimenting with Generative AI: Does ChatGPT Really Increase Everyone's Productivity?","authors":"Voraprapa Nakavachara, Tanapong Potipiti, Thanee Chaiwat","doi":"arxiv-2403.01770","DOIUrl":null,"url":null,"abstract":"Generative AI technologies such as ChatGPT, Gemini, and MidJourney have made\nremarkable progress in recent years. Recent literature has documented ChatGPT's\npositive impact on productivity in areas where it has strong expertise,\nattributable to extensive training datasets, such as the English language and\nPython/SQL programming. However, there is still limited literature regarding\nChatGPT's performance in areas where its capabilities could still be further\nenhanced. This paper aims to fill this gap. We conducted an experiment in which\neconomics students were asked to perform writing analysis tasks in a\nnon-English language (specifically, Thai) and math & data analysis tasks using\na less frequently used programming package (specifically, Stata). The findings\nsuggest that, on average, participants performed better using ChatGPT in terms\nof scores and time taken to complete the tasks. However, a detailed examination\nreveals that 34% of participants saw no improvement in writing analysis tasks,\nand 42% did not improve in math & data analysis tasks when employing ChatGPT.\nFurther investigation indicated that higher-ability students, as proxied by\ntheir econometrics grades, were the ones who performed worse in writing\nanalysis tasks when using ChatGPT. We also found evidence that students with\nbetter digital skills performed better with ChatGPT. This research provides\ninsights on the impact of generative AI. Thus, stakeholders can make informed\ndecisions to implement appropriate policy frameworks or redesign educational\nsystems. It also highlights the critical role of human skills in addressing and\ncomplementing the limitations of technology.","PeriodicalId":501487,"journal":{"name":"arXiv - QuantFin - Economics","volume":null,"pages":null},"PeriodicalIF":0.0000,"publicationDate":"2024-03-04","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"arXiv - QuantFin - Economics","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/arxiv-2403.01770","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"","JCRName":"","Score":null,"Total":0}
引用次数: 0
Abstract
Generative AI technologies such as ChatGPT, Gemini, and MidJourney have made
remarkable progress in recent years. Recent literature has documented ChatGPT's
positive impact on productivity in areas where it has strong expertise,
attributable to extensive training datasets, such as the English language and
Python/SQL programming. However, there is still limited literature regarding
ChatGPT's performance in areas where its capabilities could still be further
enhanced. This paper aims to fill this gap. We conducted an experiment in which
economics students were asked to perform writing analysis tasks in a
non-English language (specifically, Thai) and math & data analysis tasks using
a less frequently used programming package (specifically, Stata). The findings
suggest that, on average, participants performed better using ChatGPT in terms
of scores and time taken to complete the tasks. However, a detailed examination
reveals that 34% of participants saw no improvement in writing analysis tasks,
and 42% did not improve in math & data analysis tasks when employing ChatGPT.
Further investigation indicated that higher-ability students, as proxied by
their econometrics grades, were the ones who performed worse in writing
analysis tasks when using ChatGPT. We also found evidence that students with
better digital skills performed better with ChatGPT. This research provides
insights on the impact of generative AI. Thus, stakeholders can make informed
decisions to implement appropriate policy frameworks or redesign educational
systems. It also highlights the critical role of human skills in addressing and
complementing the limitations of technology.