Isolating the effect of rater experience as a time‐variant predictor of performance ratings

Abstract

A defining but sometimes overlooked characteristic of performance appraisals is that they are cyclical. The cyclical nature of performance appraisals makes it important to consider time‐variant definitions and operationalizations of constructs such as rater experience. In the current study, we work to clarify the association between rater experience and performance ratings by operationalizing rater experience as the number of appraisal cycles raters participated in. We did so while controlling for other similar but distinct operationalizations of experience such as span of control (number of ratees per rater) and familiarity with ratees. Furthermore, we employed a multilevel longitudinal design and analysis that allowed us to model rater experience as a time‐variant predictor of performance ratings and isolate its effects from both between‐rater and organizational context effects. The data were real appraisal data from a large South American company that contained 9233 ratees, across five appraisal cycles from 893 raters in 29 different business units, resulting in 24,608 observations. Our results revealed that rater experience had a small but statistically significant positive association with performance ratings. We also found that familiarity and span of control, were positively and negatively associated with performance ratings, respectively. Implications for practice and research are discussed.