Time-dependent metrics to assess performance prediction systems

Abstract

PurposeThe goal is to assess performance prediction systems (PPS) that are used to assist at-risk learners.Design/methodology/approachThe authors propose time-dependent metrics including earliness and stability. The authors investigate the relationships between the various temporal metrics and the precision metrics in order to identify the key earliness points in the prediction process. Authors propose an algorithm for computing earliness. Furthermore, the authors propose using an earliness-stability score (ESS) to investigate the relationship between the earliness of a classifier and its stability. The ESS is used to examine the trade-off between only time-dependent metrics. The aim is to compare its use to the earliness-accuracy score (EAS).FindingsStability and accuracy are proportional when the system's accuracy increases or decreases over time. However, when the accuracy stagnates or varies slightly, the system's stability is decreasing rather than stagnating. As a result, the use of ESS and EAS is complementary and allows for a better definition of the point of earliness in time by studying the relation-ship between earliness and accuracy on the one hand and earliness and stability on the other.Originality/valueWhen evaluating the performance of PPS, the temporal dimension is an important factor that is overlooked by traditional measures current metrics are not well suited to assessing PPS’s ability to predict correctly at the earliest, as well as monitoring predictions stability and evolution over time. Thus, in this work, the authors propose time-dependent metrics, including earliness, stability and the trade-offs, with objective to assess PPS over time.