False Positives and Deceptive Errors in SQL Assessment: A Large-scale Analysis of Online Judge Systems

Abstract

Online Judge Systems (OJSs) play a crucial role in evaluating SQL programming skills. However, OJSs may not accurately evaluate students’ queries due to the error-detection capabilities of test sets are insufficient, resulting in false positives that can mislead students and hinder their learning. This study analyzes a large-scale OJS’s evaluation dataset and identifies more than 110,000 (1.94%) false positive queries. It also validates existing SQL error categorization and reveals a new type of logical error called deceptive error, which occurs when students construct queries that pass specific test cases but fail to solve the actual problem. This type of error has been overlooked in previous research and can provide new insights into how to improve OJSs by enhancing test cases and feedback. This study contributes to the understanding of assessment and evaluation practices and processes in programming education, particularly the contribution that OJSs make to student learning and to course, staff and institutional development. It also suggests error prevention and detection techniques that can improve the effectiveness and fairness of OJSs in programming education and competitions.