Graphical Visualization of Difficulties Predicted from interaction Logs

Abstract

Automatic detection of programmer difficulty can help programmers receive timely assistance. Aggregate statistics are often used to evaluate difficulty detection algorithms, but this paper demonstrates that a more human-centered analysis can lead to additional insights. We have developed a novel visualization tool designed to assist researchers in improving difficulty detection algorithms. Assuming that data exists from a study in which both predicted programmer difficulties and ground truth were recorded while running an online algorithm for detecting difficulties, the tool allows researchers to interactively travel through a timeline showing the correlation between values of the features used to make predictions, difficulty predictions made by the online algorithm, and ground truth. We used the tool to improve an existing online algorithm based on a study involving the development of a GUI in Java. Episodes of difficulty predicted by the previously developed algorithm were correlated with features extracted from participant logs of interaction with the programming environment and web browser. The visualizations produced from the tool contribute to a better understanding of programmer actions during periods of difficulty, help to identify specific issues with the previous prediction algorithm, and suggest potential solutions to these issues. Thus, the information gained using this novel tool can be used to improve algorithms that help developers receive assistance at appropriate times.