Scaffolded projects with automated feedback are core instructional components of many massive courses. In subjects that include programming, feedback is typically provided by test cases constructed manually by the instructor. This paper explores the effectiveness of fuzz testing, a randomized technique for verifying the behavior of programs. In particular, we apply fuzz testing to identify when a student's solution differs in behavior from a reference implementation by randomly exploring the space of legal inputs to a program. Fuzz testing serves as a useful complement to manually constructed tests. Instructors can concentrate on designing targeted tests that focus attention on specific issues while using fuzz testing for comprehensive error checking. In the first project of a 1,400-student introductory computer science course, fuzz testing caught errors that were missed by a suite of targeted test cases for more than 48% of students. As a result, the students dedicated substantially more effort to mastering the nuances of the assignment.
{"title":"Fuzz Testing Projects in Massive Courses","authors":"S. Sridhara, Brian Hou, Jeffrey Lu, John DeNero","doi":"10.1145/2876034.2876050","DOIUrl":"https://doi.org/10.1145/2876034.2876050","url":null,"abstract":"Scaffolded projects with automated feedback are core instructional components of many massive courses. In subjects that include programming, feedback is typically provided by test cases constructed manually by the instructor. This paper explores the effectiveness of fuzz testing, a randomized technique for verifying the behavior of programs. In particular, we apply fuzz testing to identify when a student's solution differs in behavior from a reference implementation by randomly exploring the space of legal inputs to a program. Fuzz testing serves as a useful complement to manually constructed tests. Instructors can concentrate on designing targeted tests that focus attention on specific issues while using fuzz testing for comprehensive error checking. In the first project of a 1,400-student introductory computer science course, fuzz testing caught errors that were missed by a suite of targeted test cases for more than 48% of students. As a result, the students dedicated substantially more effort to mastering the nuances of the assignment.","PeriodicalId":20739,"journal":{"name":"Proceedings of the Third (2016) ACM Conference on Learning @ Scale","volume":"1 1","pages":""},"PeriodicalIF":0.0,"publicationDate":"2016-04-25","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"84758329","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Information sharing is a key activity of Massive Online Open Courses (MOOCs) user behavior. Sharing Uniform Resource Locators (URLs) has been identified as a means for individuals in online spaces to generate social relationships, construct knowledge, and disseminate information; however this activity has not been investigated within the MOOC space. This paper presents an observational study of URL sharing within MOOCs, and explores how a MOOC learning community responded to this micro behaviour. The research explored 1,471 comments and 416 learners who displayed URL sharing behavior from two iterations of the "Irish Lives" Futurelearn / Trinity College, University of Dublin MOOC. The analysis identified patterns of behavior within "URL Sharers", and suggests that this activity could support greater learner interaction. Although causality is not implied, the results of this analysis contributes a tentative new understanding of URL sharing in MOOCs, and denotes a new MOOC micro behaviour. This can be useful for MOOC practitioners to facilitate design choices.
{"title":"Observing URL Sharing Behaviour in Massive Online Open Courses","authors":"S. Gallagher, T. Savage","doi":"10.1145/2876034.2893387","DOIUrl":"https://doi.org/10.1145/2876034.2893387","url":null,"abstract":"Information sharing is a key activity of Massive Online Open Courses (MOOCs) user behavior. Sharing Uniform Resource Locators (URLs) has been identified as a means for individuals in online spaces to generate social relationships, construct knowledge, and disseminate information; however this activity has not been investigated within the MOOC space. This paper presents an observational study of URL sharing within MOOCs, and explores how a MOOC learning community responded to this micro behaviour. The research explored 1,471 comments and 416 learners who displayed URL sharing behavior from two iterations of the \"Irish Lives\" Futurelearn / Trinity College, University of Dublin MOOC. The analysis identified patterns of behavior within \"URL Sharers\", and suggests that this activity could support greater learner interaction. Although causality is not implied, the results of this analysis contributes a tentative new understanding of URL sharing in MOOCs, and denotes a new MOOC micro behaviour. This can be useful for MOOC practitioners to facilitate design choices.","PeriodicalId":20739,"journal":{"name":"Proceedings of the Third (2016) ACM Conference on Learning @ Scale","volume":"21 1","pages":""},"PeriodicalIF":0.0,"publicationDate":"2016-04-25","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"75816716","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Preliminary research is presented on the generalisability of confusion, urgency and sentiment classifiers for MOOC forum posts. The Stanford MOOCPosts data set is used to train classifiers with forum posts from individual courses and validate these classifiers on MOOC forum posts from other domain areas. While low cross-domain classification accuracy is achieved, the experiment highlights the need for transfer learning and domain adaptation algorithms; and provides insight into the types of algorithms required within an educational context.
{"title":"Towards Cross-domain MOOC Forum Post Classification","authors":"Aneesha Bakharia","doi":"10.1145/2876034.2893427","DOIUrl":"https://doi.org/10.1145/2876034.2893427","url":null,"abstract":"Preliminary research is presented on the generalisability of confusion, urgency and sentiment classifiers for MOOC forum posts. The Stanford MOOCPosts data set is used to train classifiers with forum posts from individual courses and validate these classifiers on MOOC forum posts from other domain areas. While low cross-domain classification accuracy is achieved, the experiment highlights the need for transfer learning and domain adaptation algorithms; and provides insight into the types of algorithms required within an educational context.","PeriodicalId":20739,"journal":{"name":"Proceedings of the Third (2016) ACM Conference on Learning @ Scale","volume":"2 3 1","pages":""},"PeriodicalIF":0.0,"publicationDate":"2016-04-25","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"75525026","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Students in online courses generate large amounts of data that can be used to personalize the learning process and improve quality of education. In this paper, we present the Latent Skill Embedding (LSE), a probabilistic model of students and educational content that can be used to recommend personalized sequences of lessons with the goal of helping students prepare for specific assessments. Akin to collaborative filtering for recommender systems, the algorithm does not require students or content to be described by features, but it learns a representation using access traces. We formulate this problem as a regularized maximum-likelihood embedding of students, lessons, and assessments from historical student-content interactions. Empirical findings on large-scale data from Knewton, an adaptive learning technology company, show that this approach predicts assessment results competitively with benchmark models and is able to discriminate between lesson sequences that lead to mastery and failure.
{"title":"Learning Student and Content Embeddings for Personalized Lesson Sequence Recommendation","authors":"S. Reddy, I. Labutov, T. Joachims","doi":"10.1145/2876034.2893375","DOIUrl":"https://doi.org/10.1145/2876034.2893375","url":null,"abstract":"Students in online courses generate large amounts of data that can be used to personalize the learning process and improve quality of education. In this paper, we present the Latent Skill Embedding (LSE), a probabilistic model of students and educational content that can be used to recommend personalized sequences of lessons with the goal of helping students prepare for specific assessments. Akin to collaborative filtering for recommender systems, the algorithm does not require students or content to be described by features, but it learns a representation using access traces. We formulate this problem as a regularized maximum-likelihood embedding of students, lessons, and assessments from historical student-content interactions. Empirical findings on large-scale data from Knewton, an adaptive learning technology company, show that this approach predicts assessment results competitively with benchmark models and is able to discriminate between lesson sequences that lead to mastery and failure.","PeriodicalId":20739,"journal":{"name":"Proceedings of the Third (2016) ACM Conference on Learning @ Scale","volume":"108 1","pages":""},"PeriodicalIF":0.0,"publicationDate":"2016-04-25","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"80935538","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Automated grading is essential for scaling up learning. In this paper, we conduct the first systematic study of how to automate grading of a complex assignment using a medical case assessment as a test case. We propose to solve this problem using a supervised learning approach and introduce three general complementary types of feature representations of such complex assignments for use in supervised learning. We first show with empirical experiments that it is feasible to automate grading of such assignments provided that the instructor can grade a number of examples. We further study how to integrate an automated grader with human grading and propose to frame the problem as learning to rank assignments to exploit pairwise preference judgments and use NDPM as a measure for evaluation of the accuracy of ranking. We then propose a sequential pairwise online active learning strategy to minimize the effort of human grading and optimize the collaboration of human graders and an automated grader. Experiment results show that this strategy is indeed effective and can substantially reduce human effort as compared with randomly sampling assignments for manual grading.
{"title":"An Exploration of Automated Grading of Complex Assignments","authors":"Chase Geigle, ChengXiang Zhai, D. Ferguson","doi":"10.1145/2876034.2876049","DOIUrl":"https://doi.org/10.1145/2876034.2876049","url":null,"abstract":"Automated grading is essential for scaling up learning. In this paper, we conduct the first systematic study of how to automate grading of a complex assignment using a medical case assessment as a test case. We propose to solve this problem using a supervised learning approach and introduce three general complementary types of feature representations of such complex assignments for use in supervised learning. We first show with empirical experiments that it is feasible to automate grading of such assignments provided that the instructor can grade a number of examples. We further study how to integrate an automated grader with human grading and propose to frame the problem as learning to rank assignments to exploit pairwise preference judgments and use NDPM as a measure for evaluation of the accuracy of ranking. We then propose a sequential pairwise online active learning strategy to minimize the effort of human grading and optimize the collaboration of human graders and an automated grader. Experiment results show that this strategy is indeed effective and can substantially reduce human effort as compared with randomly sampling assignments for manual grading.","PeriodicalId":20739,"journal":{"name":"Proceedings of the Third (2016) ACM Conference on Learning @ Scale","volume":"42 1","pages":""},"PeriodicalIF":0.0,"publicationDate":"2016-04-25","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"90369302","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Taylor Martin, S. Brasiel, Soojeong Jeong, Kevin Close, Kevin Lawanto, Phil Janisciewcz
Digital learning environments are becoming more common for students to engage in during and outside of school. With the immense amount of data now available from these environments, researchers need tools to process, manage, and analyze the data. Current methods used by many education researchers are inefficient; however, without data science experience tools used in other professions are not accessible. In this paper, we share about a tool we created called the Functional Understanding Navigator! (FUN! Tool). We have used this tool for different research projects which has allowed us the opportunity to (1) organize our workflow process from start to finish, (2) record log data of all of our analyses, and (3) provide a platform to share our analyses with others through GitHub. This paper extends and improves existing work in educational data mining and learning analytics.
{"title":"Macro Data for Micro Learning: Developing the FUN! Tool for Automated Assessment of Learning","authors":"Taylor Martin, S. Brasiel, Soojeong Jeong, Kevin Close, Kevin Lawanto, Phil Janisciewcz","doi":"10.1145/2876034.2893422","DOIUrl":"https://doi.org/10.1145/2876034.2893422","url":null,"abstract":"Digital learning environments are becoming more common for students to engage in during and outside of school. With the immense amount of data now available from these environments, researchers need tools to process, manage, and analyze the data. Current methods used by many education researchers are inefficient; however, without data science experience tools used in other professions are not accessible. In this paper, we share about a tool we created called the Functional Understanding Navigator! (FUN! Tool). We have used this tool for different research projects which has allowed us the opportunity to (1) organize our workflow process from start to finish, (2) record log data of all of our analyses, and (3) provide a platform to share our analyses with others through GitHub. This paper extends and improves existing work in educational data mining and learning analytics.","PeriodicalId":20739,"journal":{"name":"Proceedings of the Third (2016) ACM Conference on Learning @ Scale","volume":"228 1","pages":""},"PeriodicalIF":0.0,"publicationDate":"2016-04-25","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"86712701","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Writing a thesis is no less challenging a task for students, than for organizations who instruct and tutor thesis writing at higher education institutions. Annually within just our departments, 1000 undergraduates face the task of writing a thesis. Increasing student numbers and stagnating resources pose management problems, as well as constant threats to the quality of instruction. In reaction to this, we started exploring how instruction and supervision of thesis writers and related administrative tasks could be electronically supported, allowing for scale effects. A learning environment named Thesis Writer (TW) was developed, and piloted during the fall of 2015. TW supports individual writing and collaboration between writers, peers, tutors, and supervisors. This web-based software runs in common web browsers, independently of the operating system. In this paper we highlight the core functions of TW and address such uses in which scale effects can be realized. Conference attendees can use and test the system including real-time collaboration, in either English or German, and discuss experiences made and data collected during the pilot by 300 BA students.
{"title":"Thesis Writer (TW): Tapping Scale Effects in Academic Writing Instruction","authors":"Christian Rapp, Otto Kruse","doi":"10.1145/2876034.2893400","DOIUrl":"https://doi.org/10.1145/2876034.2893400","url":null,"abstract":"Writing a thesis is no less challenging a task for students, than for organizations who instruct and tutor thesis writing at higher education institutions. Annually within just our departments, 1000 undergraduates face the task of writing a thesis. Increasing student numbers and stagnating resources pose management problems, as well as constant threats to the quality of instruction. In reaction to this, we started exploring how instruction and supervision of thesis writers and related administrative tasks could be electronically supported, allowing for scale effects. A learning environment named Thesis Writer (TW) was developed, and piloted during the fall of 2015. TW supports individual writing and collaboration between writers, peers, tutors, and supervisors. This web-based software runs in common web browsers, independently of the operating system. In this paper we highlight the core functions of TW and address such uses in which scale effects can be realized. Conference attendees can use and test the system including real-time collaboration, in either English or German, and discuss experiences made and data collected during the pilot by 300 BA students.","PeriodicalId":20739,"journal":{"name":"Proceedings of the Third (2016) ACM Conference on Learning @ Scale","volume":"41 1","pages":""},"PeriodicalIF":0.0,"publicationDate":"2016-04-25","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"88977387","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
MOOCs by their design are able to reach several thousands of participants with very few instructors creating, delivering and facilitating the content. Participants interact with each other usually with text based asynchronous discussion forums built into the MOOC platform. The purpose of this research is to explore the role of social presence in facilitating peer support among a large community of learners.
{"title":"PEER Support In MOOCs: The Role Of Social Presence","authors":"Kwamena Appiah-Kubi, Duncan Rowland","doi":"10.1145/2876034.2893423","DOIUrl":"https://doi.org/10.1145/2876034.2893423","url":null,"abstract":"MOOCs by their design are able to reach several thousands of participants with very few instructors creating, delivering and facilitating the content. Participants interact with each other usually with text based asynchronous discussion forums built into the MOOC platform. The purpose of this research is to explore the role of social presence in facilitating peer support among a large community of learners.","PeriodicalId":20739,"journal":{"name":"Proceedings of the Third (2016) ACM Conference on Learning @ Scale","volume":"73 1","pages":""},"PeriodicalIF":0.0,"publicationDate":"2016-04-25","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"80359013","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Denise C. Nacu, C. K. Martin, Michael Schutzenhofer, Nichole Pinkard
While log analysis in massively open online courses and other online learning environments has mainly focused on traditional measures, such as completion rates and views of course content, research is responding to calls for analytic frameworks that are more reflective of social learning models. We introduce a generalizable approach to automatically code log data that highlights educator support roles and student actions that are consistent with recent conceptualizations of 21st century learning, such as creative production, self-directed learning, and social learning. Here, we describe details of a log-coding framework that builds from prior mixed method studies of the use of iRemix, an online social learning network, by middle school youth and adult educators in blended learning contexts.
{"title":"Beyond Traditional Metrics: Using Automated Log Coding to Understand 21st Century Learning Online","authors":"Denise C. Nacu, C. K. Martin, Michael Schutzenhofer, Nichole Pinkard","doi":"10.1145/2876034.2893413","DOIUrl":"https://doi.org/10.1145/2876034.2893413","url":null,"abstract":"While log analysis in massively open online courses and other online learning environments has mainly focused on traditional measures, such as completion rates and views of course content, research is responding to calls for analytic frameworks that are more reflective of social learning models. We introduce a generalizable approach to automatically code log data that highlights educator support roles and student actions that are consistent with recent conceptualizations of 21st century learning, such as creative production, self-directed learning, and social learning. Here, we describe details of a log-coding framework that builds from prior mixed method studies of the use of iRemix, an online social learning network, by middle school youth and adult educators in blended learning contexts.","PeriodicalId":20739,"journal":{"name":"Proceedings of the Third (2016) ACM Conference on Learning @ Scale","volume":"74 1","pages":""},"PeriodicalIF":0.0,"publicationDate":"2016-04-25","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"85139856","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Whilst most research on MOOCs makes inferences about the experience of learners from their interaction with the platform, few considered the rich feedback provided by learners. This paper presents the application of a conceptual model of student experience borrowed from higher education. Its relevance in the context of MOOCs was tested by using a range of questions and presentation methods in four MOOCs selected for their specific features. With varying response rates, results from over 8900 participants show how universities might view and evaluate the experience in MOOCs compared with that in traditional courses.
{"title":"Evaluating the 'Student' Experience in MOOCs","authors":"L. Vigentini, Catherine Zhao","doi":"10.1145/2876034.2893469","DOIUrl":"https://doi.org/10.1145/2876034.2893469","url":null,"abstract":"Whilst most research on MOOCs makes inferences about the experience of learners from their interaction with the platform, few considered the rich feedback provided by learners. This paper presents the application of a conceptual model of student experience borrowed from higher education. Its relevance in the context of MOOCs was tested by using a range of questions and presentation methods in four MOOCs selected for their specific features. With varying response rates, results from over 8900 participants show how universities might view and evaluate the experience in MOOCs compared with that in traditional courses.","PeriodicalId":20739,"journal":{"name":"Proceedings of the Third (2016) ACM Conference on Learning @ Scale","volume":"61 1","pages":""},"PeriodicalIF":0.0,"publicationDate":"2016-04-25","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"80587522","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
京公网安备 11010802042870号
京ICP备2023020795号-1
扫码关注我们
求助内容：
应助结果提醒方式：