The Generation and Use of Intraclass Correlation in Interactive Staff Training Research

Abstract Intraclass correlation (ICC) is an underutilized statistical technique in program consultation and program development research. One potential use is to quantitatively characterize the degree of similarity or difference among staff on a number of issues, and this can be a complement to the use of single items assessing perceived similarity between staff members. In this paper, we demonstrate the use of ICCs for quantifying agreement from both dichotomous and nondichotomous staff ratings. A number of issues in the calculation and use of ICCs are noted, especially pertaining to various forms of bias. Data generated from an ongoing Interactive Staff Training project demonstrated that: 1) different programs vary widely in their degree of staff consistency in perceptions and attitudes; 2) ICCs correlate highly with trainer perceptions of staff commitment to change at the sites receiving consultation, and with perceived level of rapport between trainers and staff; and 3) in many cases ICCs correlate differently with outcome variables compared to total scores on the same measures. In addition to providing an introduction to ICCs, instructions in their generation and explanations of output from two popular statistical packages are explained.