Undergraduate and graduate students' conceptual understanding of model classification outcomes under the lens of scientific argumentation

Recent advancements in artificial intelligence (AI) and machine learning (ML) have driven research and development across multiple industries to meet national economic and technological demands. Consequently, companies are investing in AI, ML, and data analytics workforce development efforts to digitalize operations and enhance global competitiveness. As such, evidence-based educational research around ML is essential to provide a foundation for the future workforce as they face complex AI challenges. This study explored students' conceptual ML understanding through a scientific argumentation framework, where we examined how they used evidence and reasoning to support claims about their ML models. This framework lets us gain insight into students' conceptualizations and helped scaffold student learning via a cognitive apprenticeship model. Thirty students in a mechanical engineering classroom at Purdue University experimented with neural network ML models within a computational notebook to create visual claims (ML models) with textual explanations of their evidence and reasoning. Accordingly, we qualitatively analyzed their learning artifacts to examine their underfit, fit, and overfit models and explanations. It was found that some students tended toward technical explanations while others used visual explanations. Students with technically dominant explanations had higher proficiency in generating correctly fit models but lacked explanatory evidence. Conversely, students with visually dominant explanations provided evidence but lacked technical reasoning and were less accurate in identifying fit models. We discuss implications for both groups of students and offer future research directions to examine how positive pedagogical elements of learning design can optimize ML educational material and AI workforce development.