Nonphysician Evaluators and Recording-Based Tools in Surgical Skill Assessment: A Feasibility Study

Abstract

Objectives

To investigate the feasibility of nonmedically trained evaluators and image- and video-based tools in the assessment of surgical skills in a key orthopedic procedure.

Design

Orthopedic surgeons at varying skill levels were evaluated by their ability to repair a cadaveric bi-malleolar ankle fracture. Nonphysician viewers and expert orthopedic surgeons independently scored video recordings and fluoroscopy images of the procedure through Global Rating Scales (GRS) and procedure-specific checklist tools. Statistical analysis was used to determine if the evaluators and assessment tools were able to differentiate skill level.

Setting

An academic tertiary care hospital.

Participants

The surgical procedure was completed by 3 orthopedic residents, 3 orthopedic trauma fellows, and 4 orthopedic trauma attending surgeons. The procedure was independently evaluated by 2 orthopedic surgeons and 2 nonphysicians.

Results

Operating participants were stratified by ≤ or >10 bimalleolar ankle fracture cases performed alone (inexperienced, n = 5 vs experienced, n = 5). Expert surgeon viewers could effectively stratify skill group through the GRS for video and fluoroscopy analysis (p < 0.05), and the video procedure-specific checklist (p < 0.05), but not the fluoroscopy procedure-specific checklist. Nonphysician viewers generally recognized skill groupings, although with less separation than surgeon viewers. These evaluators performed the best when aided by video and fluoroscopy procedure-specific checklists. Meanwhile, breakdowns of each tool into critical zones for improvement and evaluator-independent metrics such as case experience, self-reported confidence, and surgical time also indicated some skill differentiation.

Conclusions

The feasibility of using video recordings and fluoroscopic imaging based surgical skills assessment tools in orthopedic trauma was demonstrated. The tools highlighted in this study are applicable to both cadaver laboratory settings and live surgeries. The degree of training that is required by the evaluators and the utility of measuring surgical times of specific tasks should be the subject of future studies.