Using Think-aloud Interviews to Examine a Clinically Oriented Performance Assessment Rubric

Abstract

ABSTRACT Performance-based assessment is a common approach to assess the development and acquisition of practice competencies among health professions students. Judgments related to the quality of performance are typically operationalized as ratings against success criteria specified within a rubric. The extent to which the rubric is understood, interpreted, and applied by assessors is critical to support valid score interpretations and their subsequent use. Therefore, the purpose of this study was to examine evidence to support a scoring inference related to assessor ratings on a clinically oriented performance-based examination. Think-aloud data showed that rubric dimensions generally informed assessors’ ratings, but specific performance descriptors were rarely invoked. These findings support revisions to the rubric (e.g., less subjective, rating-scale language) and highlight tensions and implications of using rubrics for student evaluation and making decisions in a learning context.