Drops and Kinks: Modeling the Retention of Flow for Hour of Code Style Tutorials

Abstract

It can be difficult to evaluate Hour of Code activities for outcome measures such as motivation. Participation levels, for example, might be more indicative of marketing effectiveness and give little insight into longitudinal user engagement. By imagining these activities as a series of steps, we can develop a survival function model based on simple Markov chains. The student-retention this model predicts can be compared to empirical retention data gathered from traditional step-by-step and puzzle based programming tutorials. Retention of Flow is an affective evaluation [1] instrument that compares empirical student retention data to this model to better understand student motivation throughout the activity and beyond. This paper discusses two specific aspects of this Retention of Flow analysis. Drops, or sharp declines in retention, indicate a loss of motivation resulting from cognitive, practical and technical challenges. Kinks in retention indicate more gradual shifts in activity motivation. This paper uses data from a puzzle and a tutorial-based Hour of Code activity to show how understanding the Retention of Flow as a mathematical model can help with the evaluation and the design of programming tutorials.