Certified to Evaluate: Exploring Administrator Accuracy and Beliefs in Teacher Observation

Abstract

The observation systems being used in all 50 states require administrators to learn to accurately and reliably score their teachers' instruction using standardized observation systems. Although the literature on observation systems is growing, relatively few studies have examined the outcomes of trainings focused on developing administrators' accuracy using observation systems and the administrators' perceptions of that training. Therefore, the focus of this study is on examining administrators' efforts to become accurate and reliable within the context of a comprehensive teacher evaluation reform. This study was conducted during the year-long training and implementation of a new observation system in the context of a large urban district's teacher evaluation reform. The study brings together data on the outcomes of the district training—results on a certification exercise from all administrators in the district—with two sources of data on administrators' perceptions and beliefs. Specifically, we collected fall and spring survey data from nearly 300 administrators and longitudinal interview data from a subsample of 24 administrators. Taken together, these data allowed us to investigate administrators' responses to training and low-stakes practice using the observation process over 1 year. At the end of initial training, administrators reported high levels of learning, particularly in domains aligned with the focus of training. Over the year, administrators reported increased facility with the routines of conducting observations, but they still expressed learning needs, many related to the content of the observation framework. However, results from the training certification test suggested lower than desired levels of accuracy and reliability; administrators regularly did not agree with each other or with master raters. The certification test results suggested that even with a significant investment in administrator learning, there was more to be learned and mastered. If we hope for teacher evaluation to lead to the types of changes in teaching and learning that reformers have envisioned, policymakers and practitioners alike will need to devote time and resources to supporting administrator learning in initial training and throughout administrator use in practice.