Professional Behavior Assessment During Residency Training: Can We Identify Outliers?

Abstract

Introduction: The American Board of Orthopaedic Surgery (ABOS) Knowledge, Skills, and Behavior (KSB) project sets up a framework for competency-based medical education for orthopaedic surgery residency training. The Behavior aspect of KSB includes use of the American Board of Orthopaedic Surgery Behavior Tool (ABOSBT) to assess 5 domains of professional behavior. The purpose of this study was to analyze the results of 2 years of ABOSBT assessments and to characterize the low score ratings.

Methods: All ABOSBT assessments from January 1, 2022, to December 31, 2023, were reviewed totaling 51,678 completed assessments with 48,648 (94%) assessments from a 360° process and 3,030 (6%) as end-of-rotation assessments.

Results: Distribution of 51,678 ABOSBT assessments with 258,390 scores demonstrated low scores (disagree or strongly disagree) in 0.9% of assessments. All 5 domains identified low scores in a small percentage at all years-in-training with the greatest number in second- and third-year residents. Comparison of scores from 360° vs. end-of-rotation requests demonstrated a higher percentage of low scores given during the 360° process (p < 0.0001). Three thousand seven hundred seven unique evaluators completed ABOSBT assessments, with attending physicians as the most frequent evaluator type (44%); inpatient nurses assessed residents with low ABOSBT scores more frequently (4.5%) than any other evaluator type. Residents with 2 or more low scores by 2 or more different evaluators were analyzed for each of the 5 domains; low scores were most frequently observed in the reliability domain.

Discussion: The ABOSBT was originally validated 5 years ago in 18 programs with 9,892 assessments; this article updates results using the ABOSBT across 95 programs assessing 2,397 residents with 3,707 evaluators. The ABOSBT demonstrates that most residents demonstrate professional behavior across 5 domains of assessment; the ABOSBT identifies residents with low scores. A strength of KSB is the ability to identify professionalism deficiencies while residents are in training and can focus on individualized educational improvement. Tracking residents with low scores on the ABOSBT assessment over time will help determine its effectiveness in identifying unprofessional behavior.