Novice Difficulties with Analyzing the Running Time of Short Pieces of Code

Abstract

This work attempts to understand how novices approach runtime analysis tasks, where the number of operations performed by a given program needs to be analyzed by looking at the code. Such tasks are commonly used by instructors in courses on data structures and algorithms. The focus of this work is on the difficulties faced by novices when approaching such tasks and how novices differ from experts in how they approach these tasks. The study involved one-on-one think-aloud interviews with five instructors of an introductory data structures and algorithms course and 14 students enrolled in that course. The interviews were analyzed using a framework introduced in this study for describing the steps needed to perform runtime analysis of simple pieces of code. The study found the interviewed experts to clearly differentiate between the task of formulating a summation describing the number of operations performed by the code and the task of solving that summation to find how many operations are done. Experts in the study also showed fluency in looking at the code from different perspectives and sometimes re-wrote the code to simplify the analysis task. On the other hand, the study found several novices to make mistakes because of not explicitly tracing the code and because of not explicitly describing the number of performed operations using a summation. Many of the novices also seemed inclined to approach the analysis of nested loops by multiplying two running times, even when that is incorrect. The study concluded with a discussion of the implications of these results on the teaching and assessment of runtime analysis.