When trying to understand student success in computer science, much of the attention has been focused on CS1, leaving follow-up courses such as CS2 less researched. Prior studies of CS2 have often taken a deductive approach by focusing on predetermined variables such as CS1 grades, the impact of different paths from CS1 to CS2, gender and race. Although this has resulted in a better insight into these variables, we wonder if there might be another way of viewing which variables affect the students' success in the course. We have therefore chosen an inductive approach to better understand what these variables might be and how they interplay. This was done by analysing 16 semi-structured interviews with students enrolled in CS2 who have another speciality than computer science. The interviews focused mainly on the students' methods for succeeding in the course, experiences of the course and programming background. Through a thematic analysis of the interviews, we found the following five main success variables for CS2: programming competence, computer literacy, opportunity to receive help, ability to help oneself and teaching. These variables can in several cases be related to the ones previously addressed, however, they can also offer a different perspective on student success in the course.When trying to understand student success in computer science, much of the attention has been focused on CS1, leaving follow-up courses such as CS2 less researched. Prior studies of CS2 have often taken a deductive approach by focusing on predetermined variables such as CS1 grades, the impact of different paths from CS1 to CS2, gender and race. Although this has resulted in a better insight into these variables, we wonder if there might be another way of viewing which variables affect the students' success in the course. We have therefore chosen an inductive approach to better understand what these variables might be and how they interplay. This was done by analysing 16 semi-structured interviews with students enrolled in CS2 who have another speciality than computer science. The interviews focused mainly on the students' methods for succeeding in the course, experiences of the course and programming background. Through a thematic analysis of the interviews, we found the following five main success variables for CS2: programming competence, computer literacy, opportunity to receive help, ability to help oneself and teaching. These variables can in several cases be related to the ones previously addressed, however, they can also offer a different perspective on student success in the course.
{"title":"Variables Affecting Students' Success in CS2","authors":"Camilla Björn, V. Kann","doi":"10.1145/3587102.3588856","DOIUrl":"https://doi.org/10.1145/3587102.3588856","url":null,"abstract":"When trying to understand student success in computer science, much of the attention has been focused on CS1, leaving follow-up courses such as CS2 less researched. Prior studies of CS2 have often taken a deductive approach by focusing on predetermined variables such as CS1 grades, the impact of different paths from CS1 to CS2, gender and race. Although this has resulted in a better insight into these variables, we wonder if there might be another way of viewing which variables affect the students' success in the course. We have therefore chosen an inductive approach to better understand what these variables might be and how they interplay. This was done by analysing 16 semi-structured interviews with students enrolled in CS2 who have another speciality than computer science. The interviews focused mainly on the students' methods for succeeding in the course, experiences of the course and programming background. Through a thematic analysis of the interviews, we found the following five main success variables for CS2: programming competence, computer literacy, opportunity to receive help, ability to help oneself and teaching. These variables can in several cases be related to the ones previously addressed, however, they can also offer a different perspective on student success in the course.When trying to understand student success in computer science, much of the attention has been focused on CS1, leaving follow-up courses such as CS2 less researched. Prior studies of CS2 have often taken a deductive approach by focusing on predetermined variables such as CS1 grades, the impact of different paths from CS1 to CS2, gender and race. Although this has resulted in a better insight into these variables, we wonder if there might be another way of viewing which variables affect the students' success in the course. We have therefore chosen an inductive approach to better understand what these variables might be and how they interplay. This was done by analysing 16 semi-structured interviews with students enrolled in CS2 who have another speciality than computer science. The interviews focused mainly on the students' methods for succeeding in the course, experiences of the course and programming background. Through a thematic analysis of the interviews, we found the following five main success variables for CS2: programming competence, computer literacy, opportunity to receive help, ability to help oneself and teaching. These variables can in several cases be related to the ones previously addressed, however, they can also offer a different perspective on student success in the course.","PeriodicalId":410890,"journal":{"name":"Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1","volume":"80 8 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-06-29","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"124712545","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
T. Crick, T. Prickett, C. Vasiliou, Neeranjan Chitare, I. Watson
The COVID-19 pandemic has caused both significant disruption and catalysed extensive experimentation in how education has been and may be delivered worldwide. The discipline of computing has been part of this experimentation, and significant innovations have been implemented and disseminated. Furthermore, educational provision and learner expectations may have evolved in response to the experiences during the pandemic; whilst still subject to disruption, higher education in many jurisdictions is still returning to a new (ab)normal. In the UK, whilst there has been a return to traditional face-to-face delivery, one estimate suggests that nearly a third of courses continue to involve some hybrid learning as opposed to about 5% before the pandemic. Learners' preferences for delivery approach remains critical and has been explored in previous work predominately by survey-based approach. This single UK institutional case study explores learner preferences by using workshops (N=45 students across ten workshops) which aspires to co-create an understanding of issues and benefits of the learning, teaching and assessment approaches adopted in the pandemic and identify preferences for future directions. The workshops employed the sailboat retrospective template; they suggest that the preferences are varied but commonly include some face-to-face study and that digital exclusion, social engagement, and motivation/distraction are vital factors to be addressed for further adoption of hybrid or online learning.
{"title":"Exploring Computing Students' Post-Pandemic Learning Preferences with Workshops: A UK Institutional Case Study","authors":"T. Crick, T. Prickett, C. Vasiliou, Neeranjan Chitare, I. Watson","doi":"10.1145/3587102.3588807","DOIUrl":"https://doi.org/10.1145/3587102.3588807","url":null,"abstract":"The COVID-19 pandemic has caused both significant disruption and catalysed extensive experimentation in how education has been and may be delivered worldwide. The discipline of computing has been part of this experimentation, and significant innovations have been implemented and disseminated. Furthermore, educational provision and learner expectations may have evolved in response to the experiences during the pandemic; whilst still subject to disruption, higher education in many jurisdictions is still returning to a new (ab)normal. In the UK, whilst there has been a return to traditional face-to-face delivery, one estimate suggests that nearly a third of courses continue to involve some hybrid learning as opposed to about 5% before the pandemic. Learners' preferences for delivery approach remains critical and has been explored in previous work predominately by survey-based approach. This single UK institutional case study explores learner preferences by using workshops (N=45 students across ten workshops) which aspires to co-create an understanding of issues and benefits of the learning, teaching and assessment approaches adopted in the pandemic and identify preferences for future directions. The workshops employed the sailboat retrospective template; they suggest that the preferences are varied but commonly include some face-to-face study and that digital exclusion, social engagement, and motivation/distraction are vital factors to be addressed for further adoption of hybrid or online learning.","PeriodicalId":410890,"journal":{"name":"Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1","volume":"106 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-06-29","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"128829783","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Regular expressions (regex) are a text processing method widely used in data analysis, web scraping, and input validation. However, students find regular expressions difficult to create since they use a terse language of characters. Parsons problems can be a more efficient way to practice programming than typing the equivalent code with similar learning gains. In traditional Parsons problems, learners place mixed-up fragments with one or more lines in each fragment in order to solve a problem. To investigate learning regex with Parsons problems, we introduce micro Parsons problems, in which learners assemble fragments in a single line. We conducted both a think-aloud study and a large-scale between-subjects field study to evaluate this new approach. The think-aloud study provided insights into learners' perceptions of the advantages and disadvantages of solving micro Parsons problems versus traditional text-entry problems, student preferences, and revealed design considerations for micro Parsons problems. The between-subjects field study of 3,752 participants compared micro Parsons problems with text-entry problems as an optional assignment in a MOOC. The dropout rate for the micro Parsons condition was significantly lower than the text-entry condition. No significant difference was found for the learning gain on questions testing comprehensive regex skills between the two conditions, but the micro Parsons group had a significantly higher learning gain on multiple choice questions which tested understanding of regex characters.
{"title":"Using Micro Parsons Problems to Scaffold the Learning of Regular Expressions","authors":"Zihan Wu, B. Ericson, Christopher A. Brooks","doi":"10.1145/3587102.3588853","DOIUrl":"https://doi.org/10.1145/3587102.3588853","url":null,"abstract":"Regular expressions (regex) are a text processing method widely used in data analysis, web scraping, and input validation. However, students find regular expressions difficult to create since they use a terse language of characters. Parsons problems can be a more efficient way to practice programming than typing the equivalent code with similar learning gains. In traditional Parsons problems, learners place mixed-up fragments with one or more lines in each fragment in order to solve a problem. To investigate learning regex with Parsons problems, we introduce micro Parsons problems, in which learners assemble fragments in a single line. We conducted both a think-aloud study and a large-scale between-subjects field study to evaluate this new approach. The think-aloud study provided insights into learners' perceptions of the advantages and disadvantages of solving micro Parsons problems versus traditional text-entry problems, student preferences, and revealed design considerations for micro Parsons problems. The between-subjects field study of 3,752 participants compared micro Parsons problems with text-entry problems as an optional assignment in a MOOC. The dropout rate for the micro Parsons condition was significantly lower than the text-entry condition. No significant difference was found for the learning gain on questions testing comprehensive regex skills between the two conditions, but the micro Parsons group had a significantly higher learning gain on multiple choice questions which tested understanding of regex characters.","PeriodicalId":410890,"journal":{"name":"Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1","volume":"33 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-06-29","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"122904253","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Alina Zaman, A. Cook, Vinhthuy T. Phan, Alistair Windsor
Computer science (CS) relies heavily on teaching assistants (TAs) who are often untrained in CS pedagogy. Existing research on CS TA training typically studies American undergraduate TAs at high-resource universities, ignoring the many universities that use graduate TAs, who are often international students, and that don't have the resources to implement the training strategies discussed in the literature. We describe our approach to implement graduate TA training in a high-diversity, low-resource context. We present a needs assessment, design, pilot test, and deployment of our training course, and discuss implications for other similar departments hoping to train their TAs.
{"title":"A Practical Strategy for Training Graduate CS Teaching Assistants to Provide Effective Feedback","authors":"Alina Zaman, A. Cook, Vinhthuy T. Phan, Alistair Windsor","doi":"10.1145/3587102.3588776","DOIUrl":"https://doi.org/10.1145/3587102.3588776","url":null,"abstract":"Computer science (CS) relies heavily on teaching assistants (TAs) who are often untrained in CS pedagogy. Existing research on CS TA training typically studies American undergraduate TAs at high-resource universities, ignoring the many universities that use graduate TAs, who are often international students, and that don't have the resources to implement the training strategies discussed in the literature. We describe our approach to implement graduate TA training in a high-diversity, low-resource context. We present a needs assessment, design, pilot test, and deployment of our training course, and discuss implications for other similar departments hoping to train their TAs.","PeriodicalId":410890,"journal":{"name":"Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1","volume":"66 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-06-29","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"126655113","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Similarity checking is a common approach for detecting cheating in programming courses. A known limitation is high rates of similar pairs for programs lacking variability in possible solutions, especially for small programs. We experienced this issue in our CS1 course, where similarity checking in early weeks yielded many highly-similar pairs, many of which were not likely due to copying. Yet, we wish to catch copying students early, so that we can intervene and help those students avoid developing copying habits that may cause them trouble later. Our approach is to modify the program specifications to include variability-inducing requirements, namely places in the specifications where students make choices in their solutions, where different choices reduce the similarity scores. Those variability-inducing requirements are intentionally designed to avoid making the problem much harder for students. Examples of variability-inducing requirements include adding requirements to check for invalid input, or counting items. Such requirements have many different possible ways of implementing each. Essentially, variability-inducing requirements decrease the odds that two students would submit programs scored as highly-similar by a similarity checker, even for small programs. For 5 programs in our CS1 course, we added some variability-inducing requirements. Compared to an earlier term, the similarity checker's highly-similar-pairs rate dropped from 52% to 20% on average. Students' scores stayed the same from 98% to 96%, though time did increase from 18 min to 31 min on average. Adding such requirements helps instructors to do similarity detection and perform early interventions if desired.
{"title":"Variability-Inducing Requirements for Programs: Increasing Solution Variability for Similarity Checking","authors":"Ashley Pang, Frank Vahid","doi":"10.1145/3587102.3588855","DOIUrl":"https://doi.org/10.1145/3587102.3588855","url":null,"abstract":"Similarity checking is a common approach for detecting cheating in programming courses. A known limitation is high rates of similar pairs for programs lacking variability in possible solutions, especially for small programs. We experienced this issue in our CS1 course, where similarity checking in early weeks yielded many highly-similar pairs, many of which were not likely due to copying. Yet, we wish to catch copying students early, so that we can intervene and help those students avoid developing copying habits that may cause them trouble later. Our approach is to modify the program specifications to include variability-inducing requirements, namely places in the specifications where students make choices in their solutions, where different choices reduce the similarity scores. Those variability-inducing requirements are intentionally designed to avoid making the problem much harder for students. Examples of variability-inducing requirements include adding requirements to check for invalid input, or counting items. Such requirements have many different possible ways of implementing each. Essentially, variability-inducing requirements decrease the odds that two students would submit programs scored as highly-similar by a similarity checker, even for small programs. For 5 programs in our CS1 course, we added some variability-inducing requirements. Compared to an earlier term, the similarity checker's highly-similar-pairs rate dropped from 52% to 20% on average. Students' scores stayed the same from 98% to 96%, though time did increase from 18 min to 31 min on average. Adding such requirements helps instructors to do similarity detection and perform early interventions if desired.","PeriodicalId":410890,"journal":{"name":"Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1","volume":"17 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-06-29","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"128287364","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Marcus Messer, Noam Brown, Michael Kölling, Miaojing Shi
Research into automated grading has increased as Computer Science courses grow. Dynamic and static approaches are typically used to implement these graders, the most common implementation being unit testing to grade correctness. This paper expands upon an ongoing systematic literature review to provide an in-depth analysis of how machine learning (ML) has been used to grade and give feedback on programming assignments. We conducted a backward snowball search using the ML papers from an ongoing systematic review and selected 27 papers that met our inclusion criteria. After selecting our papers, we analysed the skills graded, the preprocessing steps, the ML implementation, and the models' evaluations. We find that most the models are implemented using neural network-based approaches, with most implementing some form of recurrent neural network (RNN), including Long Short-Term Memory, and encoder/decoder with attention mechanisms. Some graders implement traditional ML approaches, typically focused on clustering. Most ML-based automated grading, not many use ML to evaluate maintainability, readability, and documentation, but focus on grading correctness, a problem that dynamic and static analysis techniques, such as unit testing, rule-based program repair, and comparison to models or approved solutions, have mostly resolved. However, some ML-based tools, including those for assessing graphical output, have evaluated the correctness of assignments that conventional implementations cannot.
{"title":"Machine Learning-Based Automated Grading and Feedback Tools for Programming: A Meta-Analysis","authors":"Marcus Messer, Noam Brown, Michael Kölling, Miaojing Shi","doi":"10.1145/3587102.3588822","DOIUrl":"https://doi.org/10.1145/3587102.3588822","url":null,"abstract":"Research into automated grading has increased as Computer Science courses grow. Dynamic and static approaches are typically used to implement these graders, the most common implementation being unit testing to grade correctness. This paper expands upon an ongoing systematic literature review to provide an in-depth analysis of how machine learning (ML) has been used to grade and give feedback on programming assignments. We conducted a backward snowball search using the ML papers from an ongoing systematic review and selected 27 papers that met our inclusion criteria. After selecting our papers, we analysed the skills graded, the preprocessing steps, the ML implementation, and the models' evaluations. We find that most the models are implemented using neural network-based approaches, with most implementing some form of recurrent neural network (RNN), including Long Short-Term Memory, and encoder/decoder with attention mechanisms. Some graders implement traditional ML approaches, typically focused on clustering. Most ML-based automated grading, not many use ML to evaluate maintainability, readability, and documentation, but focus on grading correctness, a problem that dynamic and static analysis techniques, such as unit testing, rule-based program repair, and comparison to models or approved solutions, have mostly resolved. However, some ML-based tools, including those for assessing graphical output, have evaluated the correctness of assignments that conventional implementations cannot.","PeriodicalId":410890,"journal":{"name":"Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1","volume":"43 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-06-29","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"132882306","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Lydia Kraus, Valdemar Švábenský, M. Horák, Vashek Matyás, Jan Vykopal, Pavel Čeleda
As cyber threats endanger everyone, from regular users to computing professionals, spreading cybersecurity awareness becomes increasingly critical. Therefore, our university designed an innovative cybersecurity awareness course that is freely available online for students, employees, and the general public. The course offers simple, actionable steps that anyone can use to implement defensive countermeasures. Compared to other resources, the course not only suggests learners what to do, but explains why and how to do it. To measure the course impact, we administered it to 138 computer science undergraduates within a compulsory information security and cryptography course. They completed the course as a part of their homework and filled out a questionnaire after each lesson. Analysis of the questionnaire responses revealed that the students valued the course highly. They reported new learning, perspective changes, and transfer to practice. Moreover, they suggested suitable improvements to the course. Based on the results, we have distilled specific insights to help security educators design similar courses. Lessons learned from this study are relevant for cybersecurity instructors, course designers, and educational managers.
{"title":"Want to Raise Cybersecurity Awareness? Start with Future IT Professionals.","authors":"Lydia Kraus, Valdemar Švábenský, M. Horák, Vashek Matyás, Jan Vykopal, Pavel Čeleda","doi":"10.1145/3587102.3588862","DOIUrl":"https://doi.org/10.1145/3587102.3588862","url":null,"abstract":"As cyber threats endanger everyone, from regular users to computing professionals, spreading cybersecurity awareness becomes increasingly critical. Therefore, our university designed an innovative cybersecurity awareness course that is freely available online for students, employees, and the general public. The course offers simple, actionable steps that anyone can use to implement defensive countermeasures. Compared to other resources, the course not only suggests learners what to do, but explains why and how to do it. To measure the course impact, we administered it to 138 computer science undergraduates within a compulsory information security and cryptography course. They completed the course as a part of their homework and filled out a questionnaire after each lesson. Analysis of the questionnaire responses revealed that the students valued the course highly. They reported new learning, perspective changes, and transfer to practice. Moreover, they suggested suitable improvements to the course. Based on the results, we have distilled specific insights to help security educators design similar courses. Lessons learned from this study are relevant for cybersecurity instructors, course designers, and educational managers.","PeriodicalId":410890,"journal":{"name":"Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1","volume":"30 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-06-29","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"128197375","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Innovation and technology in computer science education is driven by research and practice. Both of these activities involve the gathering and analysis of data in order to develop new tools, methods including software, or strategies to solve recent challenges in the field. However, data as basis for any new solution is hardly shared, reused and recognized. This is due to the fact that the publication of research data encompasses a number of challenges for researchers, while benefits of publishing data remain low. As a result, further analyses of data as part of secondary research are uncommon in the computer science education community. Therefore, the authors of this position paper critically reflect on current practices related to the publication of research data in this community. Moreover, a path forward is outlined for future conferences, such as ITiCSE, to become increasingly FAIR, and open with regard to research data.
{"title":"Why We Need Open Data in Computer Science Education Research","authors":"Natalie Kiesler, D. Schiffner","doi":"10.1145/3587102.3588860","DOIUrl":"https://doi.org/10.1145/3587102.3588860","url":null,"abstract":"Innovation and technology in computer science education is driven by research and practice. Both of these activities involve the gathering and analysis of data in order to develop new tools, methods including software, or strategies to solve recent challenges in the field. However, data as basis for any new solution is hardly shared, reused and recognized. This is due to the fact that the publication of research data encompasses a number of challenges for researchers, while benefits of publishing data remain low. As a result, further analyses of data as part of secondary research are uncommon in the computer science education community. Therefore, the authors of this position paper critically reflect on current practices related to the publication of research data in this community. Moreover, a path forward is outlined for future conferences, such as ITiCSE, to become increasingly FAIR, and open with regard to research data.","PeriodicalId":410890,"journal":{"name":"Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1","volume":"101 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-06-29","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"132511829","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Rishabh Balse, Bharath Valaboju, Shreya Singhal, J. Warriem, Prajish Prasad
Recent advances in artificial intelligence have led to the development of large language models (LLMs), which are able to generate text, images, and source code based on prompts provided by humans. In this paper, we explore the capabilities of an LLM - OpenAI's GPT-3 model to provide feedback for student written code. Specifically, we examine the feasibility of GPT-3 to check, critique and suggest changes to code written by learners in an online programming exam of an undergraduate Python programming course. We collected 1211 student code submissions from 7 questions asked in a programming exam, and provided the GPT-3 model with separate prompts to check, critique and provide suggestions on these submissions. We found that there was a high variability in the accuracy of the model's feedback for student submissions. Across questions, the range for accurately checking the correctness of the code was between 57% to 79%, between 41% to 77% for accurately critiquing code, and between 32% and 93% for suggesting appropriate changes to the code. We also found instances where the model generated incorrect and inconsistent feedback. These findings suggest that models like GPT-3 currently cannot be 'directly' used to provide feedback to students for programming assessments.
{"title":"Investigating the Potential of GPT-3 in Providing Feedback for Programming Assessments","authors":"Rishabh Balse, Bharath Valaboju, Shreya Singhal, J. Warriem, Prajish Prasad","doi":"10.1145/3587102.3588852","DOIUrl":"https://doi.org/10.1145/3587102.3588852","url":null,"abstract":"Recent advances in artificial intelligence have led to the development of large language models (LLMs), which are able to generate text, images, and source code based on prompts provided by humans. In this paper, we explore the capabilities of an LLM - OpenAI's GPT-3 model to provide feedback for student written code. Specifically, we examine the feasibility of GPT-3 to check, critique and suggest changes to code written by learners in an online programming exam of an undergraduate Python programming course. We collected 1211 student code submissions from 7 questions asked in a programming exam, and provided the GPT-3 model with separate prompts to check, critique and provide suggestions on these submissions. We found that there was a high variability in the accuracy of the model's feedback for student submissions. Across questions, the range for accurately checking the correctness of the code was between 57% to 79%, between 41% to 77% for accurately critiquing code, and between 32% and 93% for suggesting appropriate changes to the code. We also found instances where the model generated incorrect and inconsistent feedback. These findings suggest that models like GPT-3 currently cannot be 'directly' used to provide feedback to students for programming assessments.","PeriodicalId":410890,"journal":{"name":"Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1","volume":"14 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-06-29","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"134257811","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
In recent years Parsons problems have grown in popularity as both a pedagogical tool and as an assessment item alike. In these problems, students are expected to take existing but jumbled blocks of code and organize them to form a working solution. It is common for these problems to include incorrect blocks of code, typically referred to as "distractors," alongside the correct blocks. However, the utility of these distractors and their impact on a problems difficulty has yet to be thoroughly investigated. This study contributes to filling this gap by comparing performance, time spent, and item discrimination statistics for 32 pairs of Parsons problems from CS1 Python exams and quizzes. Our findings indicate that the inclusion of distractors has a large impact on the amount of time students spend on the questions and a low to moderate impact on score. Additionally, problems without distractors were already found to have high discrimination and including distractors did little to improve their discrimination. These findings suggest that the inclusion of distractors does little to improve the quality of these problems as exam questions but may have a negative impact on students by causing them to spend significantly more time on the problems and reducing the time they have for the rest of the exam.
{"title":"Investigating the Role and Impact of Distractors on Parsons Problems in CS1 Assessments","authors":"David H. Smith, Max Fowler, C. Zilles","doi":"10.1145/3587102.3588819","DOIUrl":"https://doi.org/10.1145/3587102.3588819","url":null,"abstract":"In recent years Parsons problems have grown in popularity as both a pedagogical tool and as an assessment item alike. In these problems, students are expected to take existing but jumbled blocks of code and organize them to form a working solution. It is common for these problems to include incorrect blocks of code, typically referred to as \"distractors,\" alongside the correct blocks. However, the utility of these distractors and their impact on a problems difficulty has yet to be thoroughly investigated. This study contributes to filling this gap by comparing performance, time spent, and item discrimination statistics for 32 pairs of Parsons problems from CS1 Python exams and quizzes. Our findings indicate that the inclusion of distractors has a large impact on the amount of time students spend on the questions and a low to moderate impact on score. Additionally, problems without distractors were already found to have high discrimination and including distractors did little to improve their discrimination. These findings suggest that the inclusion of distractors does little to improve the quality of these problems as exam questions but may have a negative impact on students by causing them to spend significantly more time on the problems and reducing the time they have for the rest of the exam.","PeriodicalId":410890,"journal":{"name":"Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1","volume":"46 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-06-29","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"134417545","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
京公网安备 11010802042870号
京ICP备2023020795号-1
扫码关注我们
求助内容：
应助结果提醒方式：