Making Hay from Wheats: A Classsourcing Method to Identify Misconceptions

Novice programmers often begin coding with a poor understanding of the task at hand and end up solving the wrong problem. A promising way to put novices on the right track is to have them write examples first, before coding, and provide them with feedback by evaluating the examples on a suite of chaff implementations that are flawed in subtle ways. This feedback, however, is only as good as the chaffs themselves. Instructors must anticipate misconceptions and avoid expert blind spots to make a useful suite of chaffs. This paper conjectures that novices’ incorrect examples are a rich source of insight and presents a classsourcing method for identifying misconceptions. First off, we identify incorrect examples using known, correct wheat implementations. The method is to manually cluster incorrect examples by semantic similarity, summarize each cluster with a potential misconception, and use the analysis to generate chaffs—thereby deriving a useful by-product (hay) from examples that fail the wheats. Classsourced misconceptions have revealed expert blind spots and drawn attention to chaffs that seldom arose in practice, one of which had an undiscovered bug.