Framing our Expectations: Variability in Entrustable Professional Activity Assessments

OBJECTIVE

To determine the ability of surgical trainees and faculty to correctly interpret entrustability of a resident learner in a modeled patient care scenario.

DESIGN

Prospective study utilizing a web-based survey including 4 previously-recorded short videos of resident learners targeted to specific levels of the American Board of Surgery's (ABS) Entrustment Scale. Respondents were asked to choose the entrustment level that best corresponded to their observations of the learner in the video. Responses were subcategorized by low and high entrustment.

SETTING

Online, utilizing the Qualtrics survey platform.

PARTICIPANTS

Survey targeting US surgical trainees and surgical faculty via email and social media. We received 31 complete responses and 2 responses which completed > 1 video assessment question without demographic information (n = 33). Respondents included 10 trainees (32%) and 21 attending surgeons (68%).

RESULTS

Neither faculty nor trainees readily identified the targeted entrustment level for Question 1 (preoperative care of a patient with acute appendicitis with high entrustment, 36% correct), though evaluations of the remaining questions (2 through 4) demonstrated more accuracy (70, 84, and 75% correct, respectively). Faculty were more readily able than trainees to identify low entrustment (level Limited Participation) in intraoperative inguinal hernia repair (95% vs 60%, p = 0.03). After subcategorization to high and low entrustment, both residents and faculty were able to accurately identify entrustment 95% overall.

CONCLUSIONS

Both trainees and attending surgeons were able to identify high- and low-performing residents on short video demonstrations using the ABS EPA entrustment scale. This provides additional evidence in support of the need for frequent observations of EPAs to account for the variability in raters’ perceptions in addition to complexity of clinical scenarios. Frame-of-reference training via a video-based platform may also be beneficial for both residents and faculty as an ongoing EPA implementation strategy.