An Analysis of Ratings and Interrater Reliability at the American String Teachers Association’s National Orchestra Festival

Abstract

The purpose of this study was to examine the assigned ratings, interrater reliability, and possible influences of school level and instrumentation on adjudicators’ evaluations of orchestra performances at a national-level adjudicated music festival. Data consisted of the overall ratings assigned to orchestra performances (N = 55) at the 2017, 2018, and 2019 American String Teachers Association’s National Orchestra Festival (NOF). Analysis revealed that 83.64% of all participating orchestras earned a I/Superior or II/Excellent overall rating. A logistic regression model revealed that both school level and instrumentation were not significant predictors of earning a I/Superior overall rating. Individual members of each three-judge adjudication panel assigned identical ratings to 61.82% of orchestra performances and possessed a mean internal consistency of .90. Orchestra teachers can use the results of this study to weigh the educational benefits of participating in the NOF.