Trade-Offs in the Implementation of Observational Ratings Systems.

Abstract

A consensus has developed that high-quality teacher evaluation systems require multiple measures. We examine multiple measures from a large urban school district, which has included observational ratings and value-added ratings in its system since 2010. Evaluation systems that do not account for observer severity, classroom context, and other factors may yield different results from systems that do account for these factors. Choosing a simpler system involves a trade-off regarding a system's robustness or defensibility. Using a many-faceted Rasch model, we explore rating components like observer, time of year, and subdomain. We find high reliability of the resulting teacher ratings, some impact of adjusting for observer differences and differences between subdomains, and positive correlation with value-added measures. A comprehensive analysis like MFRM should be part of a district's evaluation system, even if only as a robustness check, and districts should examine how observational scores and classroom context are related.