Design and implementation of an AI-enabled visual report tool as formative assessment to promote learning achievement and self-regulated learning: An experimental study

Abstract

Formative assessment is essential for improving teaching and learning, and AI and visualization techniques provide great potential for its design and delivery. Using NLP, cognitive diagnostic and visualization techniques designed to analyse and present students' monthly exam data, we developed an AI-enabled visual report tool comprising six modules and conducted an empirical study of its effectiveness in a high school biology classroom. A total of 125 students in a ninth-grade biology course were assigned to a treatment group (n = 63) receiving AI-enabled visual reports as the intervention and a control group (n = 62) receiving overall oral feedback from the teacher. We present the main statistical results of the within-subjects design and the between-subjects design respectively, to better capture the main findings. Repeated measures ANOVA revealed a significant interaction effect of intervention and time on learning achievement, and the paired-sample Wilcoxon test indicated that the treatment group had experienced increasing learning anxiety (Cohen's d = 0.203, p = 0.046) and self-efficacy (Cohen's d = 1.793, p = 0.000) over time. Moreover, we conducted a series of non-parametric tests to compare the effects of AI-enabled visual reports and teacher feedback, but found no significant differences except for an increased self-efficacy (Cohen's d = 0.312, p = 0.046). Additionally, we had the students in the treatment group rate their favourable modules in the AI-enabled visual report and provide evaluative feedback. The study results provide important insights into the design and implementation of effective formative assessment supported by artificial AI and visualization techniques.

Practitioner notes

What is already known about this topic

• Formative assessment is essential for improving teaching and learning.
• Traditional formative assessment tools lack accurate data-oriented assessment and usability.
• AI and visualization techniques have great potential for formative assessment.

What this paper adds

• This study designs and implements an AI-enabled visual report tool that generates data-driven, user-friendly reports.
• The AI-enabled visual report can not only enhance students' learning achievement and self-regulated learning over time but also increase their test anxiety.
• The AI-enabled visual report has a comparable effect with teacher feedback but leads to increased self-efficacy.

Implications for practice and/or policy

• We recommend using the AI-enabled visual report in large-size classes for its overall positive effects on both learning achievement and self-regulated learning.
• We recommend using the AI-enabled visual report over teacher feedback for its capacity to enhance students' self-efficacy.
• We recommend prioritizing the modules of Performance Ranking, Personal Mastery and Knowledge Alert when designing the AI-enabled visual report.