Effects of observation mode on ratings of teaching quality in secondary mathematics classrooms

Abstract

In educational research, teaching quality is extensively studied because of its role of a mediator between teacher characteristics and student learning. However, empirical evidence on differences between video and live scoring of teaching quality is rare. In the present study, thirty lessons from 15 secondary mathematics classrooms in a German metropolitan area were observed. Lessons were scored both live in the classroom and using video recordings. Live and video scoring was conducted by (different) trained observers. Ratings were obtained with a “hybrid” observational instrument that covers generic and subject-specific characteristics of teaching quality in mathematics classrooms. Generalizability analysis and paired t tests were performed to investigate mode effects. The findings showed that in live scoring, classroom management was rated lower, and cognitive activation was rated higher. Rankings of lessons or classrooms were very similar across modes, and reliabilities did not differ to a meaningful extent either, except for classroom management reaching better results for live ratings. This suggests that based on the present findings, classroom observation performed with our hybrid framework of teaching quality generalizes across observation mode only under certain circumstances. Further research is necessary to better understand the relation between observation mode and teaching quality ratings, as well as the impact of the scoring procedures. We discuss the implications of our findings for educational research and practice.