The Uncertain Role of Educational Software in Remediating Student Learning: Regression Discontinuity Evidence from Three Local Education Agencies

Abstract Educators must balance the needs of students who start the school year behind grade level with their obligation to teach grade-appropriate content to all students. Educational software could help educators strike this balance by targeting content to students’ differing levels of mastery. Using a regression discontinuity design and detailed software log and administrative data, we compare two versions of an online mathematics program used by students in three education agencies. We find that although students assigned the modified curriculum did progress through content objectives more quickly than students assigned the default curriculum, they did not perform better on pre- and post-objective quizzes embedded in the software, and most never progressed far enough to reach the grade-level content. Furthermore, there was no statistically significant effect of the modified curriculum on formative test scores. These findings suggest policymakers and practitioners should exercise caution when assigning exclusively remedial content to students who start the school year behind grade level, even though this is a common feature of many math educational software programs.