Extrapolative Validity Evidence of the Anastomosis Objective Structured Assessment of Technical Skill (A-OSATS) for Robotic Ileocolic Anastomosis

OBJECTIVE

To collect validity evidence for the use of the Anastomosis Objective Structured Assessment of Technical Skills (A-OSATS) instrument, which has been developed to evaluate performance of a minimally invasive side-to-side bowel anastomosis with hand-sewn common enterotomy.

DESIGN

Residents performed a robotic ileocolic anastomosis simulation on an ex vivo porcine model. Faculty scored each resident with the A-OSATS and performed a provocative leak test on the completed anastomoses. Residents were reassessed on the sewing sub-score 1 month later. Data were compared with parametric and nonparametric analysis.

SETTING

Single academic general surgery residency

PARTICIPANTS

PGY-4 and -5 general surgery residents (n = 17)

RESULTS

PGY-5s performed better than PGY-4s in repeat A-OSATS sewing sub-score (mean 55/55 ± 0 vs 43 ± 4.9, p < 0.001) and time to complete (minutes, mean 14.5 ± 4.9 vs 21.2 ± 3.9, p = 0.01). There was a strong correlation between A-OSATS score and time (r = −0.67, p = 0.005). For the initial assessment, there was no significant difference in mean A-OSATS score between anastomoses that leaked and those that did not leak (137.3 ± 14.5 vs 150.1 ± 11.2, p = 0.098), but on repeat assessment, intact anastomoses had a higher mean A-OSATS sewing sub-score than those that leaked (52.2 ± 4.7 vs 39 ± 3.5, p = 0.007). There was no significant difference between initial A-OSATS score and repeat score (p = 0.14).

CONCLUSIONS

We provide extrapolative validity evidence for the A-OSATS instrument by comparing A-OSATS score to time to sew, provocative leak test, and discrimination between PGY-4s and PGY-5s. Generalizability validity evidence is provided by test-retest reliability. Further refinement is needed for the A-OSATS tool to be used for high-stakes entrustment decisions in resident-performed robotic ileocolic anastomoses.