Along with the advent of MOOCs and other online learning platforms such as Khan Academy, the role of online education has continued to grow in relation to that of traditional on-campus instruction. Rather than tackle the problem of evaluating large educational units such as entire online courses, this paper approaches a smaller problem: exploring a framework for evaluating more granular educational units, in this case, short educational videos. We have chosen to leverage an adaptation of traditional Bayesian Knowledge Tracing (BKT), intended to incorporate the usage of video content in addition to assessment activity. By exploring the change in predictive error when alternately including or omitting video activity, we suggest a metric for determining the relevance of videos to associated assessments. To validate our hypothesis and demonstrate the application of our proposed methods we use data obtained from the popular Khan Academy website.
{"title":"Toward the Evaluation of Educational Videos using Bayesian Knowledge Tracing and Big Data","authors":"Zachary MacHardy, Z. Pardos","doi":"10.1145/2724660.2728690","DOIUrl":"https://doi.org/10.1145/2724660.2728690","url":null,"abstract":"Along with the advent of MOOCs and other online learning platforms such as Khan Academy, the role of online education has continued to grow in relation to that of traditional on-campus instruction. Rather than tackle the problem of evaluating large educational units such as entire online courses, this paper approaches a smaller problem: exploring a framework for evaluating more granular educational units, in this case, short educational videos. We have chosen to leverage an adaptation of traditional Bayesian Knowledge Tracing (BKT), intended to incorporate the usage of video content in addition to assessment activity. By exploring the change in predictive error when alternately including or omitting video activity, we suggest a metric for determining the relevance of videos to associated assessments. To validate our hypothesis and demonstrate the application of our proposed methods we use data obtained from the popular Khan Academy website.","PeriodicalId":20664,"journal":{"name":"Proceedings of the Second (2015) ACM Conference on Learning @ Scale","volume":"28 1","pages":""},"PeriodicalIF":0.0,"publicationDate":"2015-03-14","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"74987043","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Formative writing systems with automated scoring provide opportunities for students to write, receive feedback, and then revise essays in a timely iterative cycle. This paper describes ongoing investigations of a formative writing tool through mining student data in order to understand how the system performs and to measure improvement in student writing. The sampled data included over 1.3M student essays written in response to approximately 200 pre-defined prompts as well as a log of all student actions and computer generated feedback. Analyses both measured and modeled changes in student performance over revisions, the effects of system responses and the amount of time students spent working on assignments. Implications are discussed for employing large-scale data analytics to improve educational outcomes, to understand the role of feedback in writing, to drive improvements in formative technology and to aid in designing better kinds of feedback and scaffolding to support students in the writing process.
{"title":"Analysis of a Large-Scale Formative Writing Assessment System with Automated Feedback","authors":"P. Foltz, Mark Rosenstein","doi":"10.1145/2724660.2728688","DOIUrl":"https://doi.org/10.1145/2724660.2728688","url":null,"abstract":"Formative writing systems with automated scoring provide opportunities for students to write, receive feedback, and then revise essays in a timely iterative cycle. This paper describes ongoing investigations of a formative writing tool through mining student data in order to understand how the system performs and to measure improvement in student writing. The sampled data included over 1.3M student essays written in response to approximately 200 pre-defined prompts as well as a log of all student actions and computer generated feedback. Analyses both measured and modeled changes in student performance over revisions, the effects of system responses and the amount of time students spent working on assignments. Implications are discussed for employing large-scale data analytics to improve educational outcomes, to understand the role of feedback in writing, to drive improvements in formative technology and to aid in designing better kinds of feedback and scaffolding to support students in the writing process.","PeriodicalId":20664,"journal":{"name":"Proceedings of the Second (2015) ACM Conference on Learning @ Scale","volume":"1 1","pages":""},"PeriodicalIF":0.0,"publicationDate":"2015-03-14","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"73168924","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
We describe a new method to troubleshoot and improve domain and student models from interactive learning environments. The method applies as long as the models can generate predictions of student behavior. The method is a visualization of model predictions, categorized using a metric of recent performance. We describe the method, its application in prior work to student models, and a proposed extension to domain models.
{"title":"Viz-R: Using Recency to Improve Student and Domain Models","authors":"Ilya M. Goldin, April Galyardt","doi":"10.1145/2724660.2728706","DOIUrl":"https://doi.org/10.1145/2724660.2728706","url":null,"abstract":"We describe a new method to troubleshoot and improve domain and student models from interactive learning environments. The method applies as long as the models can generate predictions of student behavior. The method is a visualization of model predictions, categorized using a metric of recent performance. We describe the method, its application in prior work to student models, and a proposed extension to domain models.","PeriodicalId":20664,"journal":{"name":"Proceedings of the Second (2015) ACM Conference on Learning @ Scale","volume":"260 1","pages":""},"PeriodicalIF":0.0,"publicationDate":"2015-03-14","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"76396563","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Yuanyuan Wang, Yukiko Kawai, Setsuko Miyamoto, K. Sumiya
Nowadays, many universities utilize groupware support for students to post and share their e-reports, and the students can browse and vote other students' reports in e-learning. Teachers then need to evaluate all students' reports, but this will require a great deal of time and effort for a fair evaluation of the reports. Therefore, we propose an e-report scoring method based on student peer evaluation by considering the relationship between voting and posting time of the e-reports, to promote the quality of the votes and prevent unfair votes. Then, the method can provide scores of reports based on a voting graph by analyzing students who vote the reports. In this paper, we perform a student peer evaluation using groupware based on voting with a "Like" button in a course practice, and compare the results with teachers' evaluation.
{"title":"An E-Report Scoring Method based on Student Peer Evaluation using Groupware","authors":"Yuanyuan Wang, Yukiko Kawai, Setsuko Miyamoto, K. Sumiya","doi":"10.1145/2724660.2728674","DOIUrl":"https://doi.org/10.1145/2724660.2728674","url":null,"abstract":"Nowadays, many universities utilize groupware support for students to post and share their e-reports, and the students can browse and vote other students' reports in e-learning. Teachers then need to evaluate all students' reports, but this will require a great deal of time and effort for a fair evaluation of the reports. Therefore, we propose an e-report scoring method based on student peer evaluation by considering the relationship between voting and posting time of the e-reports, to promote the quality of the votes and prevent unfair votes. Then, the method can provide scores of reports based on a voting graph by analyzing students who vote the reports. In this paper, we perform a student peer evaluation using groupware based on voting with a \"Like\" button in a course practice, and compare the results with teachers' evaluation.","PeriodicalId":20664,"journal":{"name":"Proceedings of the Second (2015) ACM Conference on Learning @ Scale","volume":"1 1","pages":""},"PeriodicalIF":0.0,"publicationDate":"2015-03-14","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"80504608","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Ruth Wylie, E. Finn, J. Eschrich, Kiyash Monsef, R. Hawkins
Designing educational games that meet both learning and entertainment objectives is a challenging task. Games that begin by developing specific educational goals and are later wrapped in a game or narrative context risk appearing forced, while those that begin with gaming elements to which educational elements are added may appear superficial. In this paper, we describe the methodology and results from a three-day interdisciplinary hackathon for developing game narratives designed to address both needs. We present details regarding the hackathon, the collaborative teams, and an example of the outcomes produced.
{"title":"Exploring Collaborative Storytelling as a Method for Creating Educational Games","authors":"Ruth Wylie, E. Finn, J. Eschrich, Kiyash Monsef, R. Hawkins","doi":"10.1145/2724660.2728692","DOIUrl":"https://doi.org/10.1145/2724660.2728692","url":null,"abstract":"Designing educational games that meet both learning and entertainment objectives is a challenging task. Games that begin by developing specific educational goals and are later wrapped in a game or narrative context risk appearing forced, while those that begin with gaming elements to which educational elements are added may appear superficial. In this paper, we describe the methodology and results from a three-day interdisciplinary hackathon for developing game narratives designed to address both needs. We present details regarding the hackathon, the collaborative teams, and an example of the outcomes produced.","PeriodicalId":20664,"journal":{"name":"Proceedings of the Second (2015) ACM Conference on Learning @ Scale","volume":"192 1 1","pages":""},"PeriodicalIF":0.0,"publicationDate":"2015-03-14","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"78320939","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
It is our great pleasure to welcome you to ACM conference Learning at Scale 2015. In this, the second year of the conference, we have seen a significant growth in the number of submissions to the conference and an overall improvement in the quality of the contributions. This year's conference continues the tradition of being the premier forum for presentation of research results and inside stories about what makes online educational systems operate at scale. The call for papers attracted submissions from all over the world, covering a broad range of topics from the theoretical to the pragmatic. The program committee reviewed and accepted the following: Venue or Track Reviewed Accepted Full Technical Papers 90 23 25% Short Technical Papers 12 5 41% Work in Progress Papers 54 47 80% Since the conference is still in its formative years, we accepted a large fraction of all the Works in Progress because we found the experience of reading through them to be so valuable. We are still a nascent field, and learning about the very latest work reflects the rapidly changing nature of what we know to be true. We encourage attendees to attend both keynotes. These valuable and insightful talks can and will guide us to a better understanding of the future of our field: Achieving 96% mastery at national scale through inspired learning and generative adaptivity, Zoran Popovic (University of Washington) Machine Learning for Learning at Scale, Peter Norvig (Google)
{"title":"Proceedings of the Second (2015) ACM Conference on Learning @ Scale","authors":"G. Kiczales, D. Russell, B. Woolf","doi":"10.1145/2724660","DOIUrl":"https://doi.org/10.1145/2724660","url":null,"abstract":"It is our great pleasure to welcome you to ACM conference Learning at Scale 2015. In this, the second year of the conference, we have seen a significant growth in the number of submissions to the conference and an overall improvement in the quality of the contributions. This year's conference continues the tradition of being the premier forum for presentation of research results and inside stories about what makes online educational systems operate at scale. \u0000 \u0000The call for papers attracted submissions from all over the world, covering a broad range of topics from the theoretical to the pragmatic. \u0000 \u0000The program committee reviewed and accepted the following: Venue or Track Reviewed Accepted \u0000Full Technical Papers 90 23 25% \u0000Short Technical Papers 12 5 41% \u0000Work in Progress Papers 54 47 80% \u0000 \u0000 \u0000 \u0000Since the conference is still in its formative years, we accepted a large fraction of all the Works in Progress because we found the experience of reading through them to be so valuable. We are still a nascent field, and learning about the very latest work reflects the rapidly changing nature of what we know to be true. \u0000 \u0000We encourage attendees to attend both keynotes. These valuable and insightful talks can and will guide us to a better understanding of the future of our field: \u0000Achieving 96% mastery at national scale through inspired learning and generative adaptivity, Zoran Popovic (University of Washington) \u0000Machine Learning for Learning at Scale, Peter Norvig (Google)","PeriodicalId":20664,"journal":{"name":"Proceedings of the Second (2015) ACM Conference on Learning @ Scale","volume":"379 1","pages":""},"PeriodicalIF":0.0,"publicationDate":"2015-03-14","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"77886811","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Min Hyung Lee, Joe Runde, Warfa Jibril, Zhuoying Wang, E. Brunskill
As a step towards scaling personalized instruction, we seek to automatically identify the key features of the interactive learning process teachers use to select the next activity when teaching a single student. Such features could both inform computational student models designed to facilitate instructional decisions, and help enable automated self-improving teaching systems that leverage this identified feature set. We present preliminary results that a very small set of features is almost as good as a much larger set of features at predicting human tutor decisions when teaching students about histograms.
{"title":"Learning the Features Used To Decide How to Teach","authors":"Min Hyung Lee, Joe Runde, Warfa Jibril, Zhuoying Wang, E. Brunskill","doi":"10.1145/2724660.2728707","DOIUrl":"https://doi.org/10.1145/2724660.2728707","url":null,"abstract":"As a step towards scaling personalized instruction, we seek to automatically identify the key features of the interactive learning process teachers use to select the next activity when teaching a single student. Such features could both inform computational student models designed to facilitate instructional decisions, and help enable automated self-improving teaching systems that leverage this identified feature set. We present preliminary results that a very small set of features is almost as good as a much larger set of features at predicting human tutor decisions when teaching students about histograms.","PeriodicalId":20664,"journal":{"name":"Proceedings of the Second (2015) ACM Conference on Learning @ Scale","volume":"1 1","pages":""},"PeriodicalIF":0.0,"publicationDate":"2015-03-14","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"83210658","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Korinn S. Ostrow, Christopher Donnelly, Seth A. Adjei, N. Heffernan
Student modeling within intelligent tutoring systems is a task largely driven by binary models that predict student knowledge or next problem correctness (i.e., Knowledge Tracing (KT)). However, using a binary construct for student assessment often causes researchers to overlook the feedback innate to these platforms. The present study considers a novel method of tabling an algorithmically determined partial credit score and problem difficulty bin for each student's current problem to predict both binary and partial next problem correctness. This study was conducted using log files from ASSISTments, an adaptive mathematics tutor, from the 2012-2013 school year. The dataset consisted of 338,297 problem logs linked to 15,253 unique student identification numbers. Findings suggest that an efficiently tabled model considering partial credit and problem difficulty performs about as well as KT on binary predictions of next problem correctness. This method provides the groundwork for modifying KT in an attempt to optimize student modeling.
{"title":"Improving Student Modeling Through Partial Credit and Problem Difficulty","authors":"Korinn S. Ostrow, Christopher Donnelly, Seth A. Adjei, N. Heffernan","doi":"10.1145/2724660.2724667","DOIUrl":"https://doi.org/10.1145/2724660.2724667","url":null,"abstract":"Student modeling within intelligent tutoring systems is a task largely driven by binary models that predict student knowledge or next problem correctness (i.e., Knowledge Tracing (KT)). However, using a binary construct for student assessment often causes researchers to overlook the feedback innate to these platforms. The present study considers a novel method of tabling an algorithmically determined partial credit score and problem difficulty bin for each student's current problem to predict both binary and partial next problem correctness. This study was conducted using log files from ASSISTments, an adaptive mathematics tutor, from the 2012-2013 school year. The dataset consisted of 338,297 problem logs linked to 15,253 unique student identification numbers. Findings suggest that an efficiently tabled model considering partial credit and problem difficulty performs about as well as KT on binary predictions of next problem correctness. This method provides the groundwork for modifying KT in an attempt to optimize student modeling.","PeriodicalId":20664,"journal":{"name":"Proceedings of the Second (2015) ACM Conference on Learning @ Scale","volume":"19 1","pages":""},"PeriodicalIF":0.0,"publicationDate":"2015-03-14","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"88405438","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Vineet Pandey, Yasmine Kotturi, Chinmay Kulkarni, Michael S. Bernstein, Scott R. Klemmer
Student discussions over video in massive classes allow students to explore course content, share personal experiences and get feedback on their ideas. However, such discussions frequently turn into casual conversations without focusing on the curriculum and the learning objectives. This short paper explores whether students can achieve multiple learning objectives by solving challenges collaboratively during discussions. We introduce `think-pair-share' technique for video discussions. Our pilot results, drawn from a Coursera class, suggest that participants prefer to exchange information with their peers using personal stories and connecting stories with curriculum increases participant engagement.
{"title":"Connecting Stories and Pedagogy Increases Participant Engagement in Discussions","authors":"Vineet Pandey, Yasmine Kotturi, Chinmay Kulkarni, Michael S. Bernstein, Scott R. Klemmer","doi":"10.1145/2724660.2728670","DOIUrl":"https://doi.org/10.1145/2724660.2728670","url":null,"abstract":"Student discussions over video in massive classes allow students to explore course content, share personal experiences and get feedback on their ideas. However, such discussions frequently turn into casual conversations without focusing on the curriculum and the learning objectives. This short paper explores whether students can achieve multiple learning objectives by solving challenges collaboratively during discussions. We introduce `think-pair-share' technique for video discussions. Our pilot results, drawn from a Coursera class, suggest that participants prefer to exchange information with their peers using personal stories and connecting stories with curriculum increases participant engagement.","PeriodicalId":20664,"journal":{"name":"Proceedings of the Second (2015) ACM Conference on Learning @ Scale","volume":"1 1","pages":""},"PeriodicalIF":0.0,"publicationDate":"2015-03-14","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"88600708","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
J. Williams, Korinn S. Ostrow, Xiaolu Xiong, Elena L. Glassman, Juho Kim, Samuel G. Maldonado, Na Li, J. Reich, N. Heffernan
In contrast to typical laboratory experiments, the everyday use of online educational resources by large populations and the prevalence of software infrastructure for A/B testing leads us to consider how platforms can embed in vivo experiments that do not merely support research, but ensure practical improvements to their educational components. Examples are presented of randomized experimental comparisons conducted by subsets of the authors in three widely used online educational platforms -- Khan Academy, edX, and ASSISTments. We suggest design principles for platform technology to support randomized experiments that lead to practical improvements -- enabling Iterative Improvement and Collaborative Work -- and explain the benefit of their implementation by WPI co-authors in the ASSISTments platform.
{"title":"Using and Designing Platforms for In Vivo Educational Experiments","authors":"J. Williams, Korinn S. Ostrow, Xiaolu Xiong, Elena L. Glassman, Juho Kim, Samuel G. Maldonado, Na Li, J. Reich, N. Heffernan","doi":"10.1145/2724660.2728704","DOIUrl":"https://doi.org/10.1145/2724660.2728704","url":null,"abstract":"In contrast to typical laboratory experiments, the everyday use of online educational resources by large populations and the prevalence of software infrastructure for A/B testing leads us to consider how platforms can embed in vivo experiments that do not merely support research, but ensure practical improvements to their educational components. Examples are presented of randomized experimental comparisons conducted by subsets of the authors in three widely used online educational platforms -- Khan Academy, edX, and ASSISTments. We suggest design principles for platform technology to support randomized experiments that lead to practical improvements -- enabling Iterative Improvement and Collaborative Work -- and explain the benefit of their implementation by WPI co-authors in the ASSISTments platform.","PeriodicalId":20664,"journal":{"name":"Proceedings of the Second (2015) ACM Conference on Learning @ Scale","volume":"29 1","pages":""},"PeriodicalIF":0.0,"publicationDate":"2015-02-14","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"88295440","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
京公网安备 11010802042870号
京ICP备2023020795号-1
扫码关注我们
求助内容：
应助结果提醒方式：