SimCapture app video performance assessment versus real-time instructor-based performance evaluation of undergraduates in neonatal resuscitation-an agreement study.

Abstract

Undergraduates are trained in the basic neonatal resuscitation programme (NRP) and evaluated for skill acquisition by NRP performance evaluation test (PET). Video use improves the validity of assessment when video-information adequacy, intrusiveness to students, and educational purpose are adequately balanced. We evaluated whether there was a difference between instructor's real-time assessment and video-based assessment done by another independent assessor using videos recorded via the SimCapture application. Undergraduates were trained in basic neonatal resuscitation and were evaluated at the end for skill acquisition by PET as a part of a randomized control trial (RCT). Video recording of the PET assessment was done with the SimCapture app. Independent assessors evaluated recorded videos to evaluate and validate real-time instructor performance evaluation time assessments. We conducted an agreement study to evaluate the difference between a real-time instructor and video assessments. We trained 143 students; 139 videos were evaluated. The Bland-Altman plot showed good agreement. For the post-test result, 66 (47.5%) passed, and 73 (52.5%) failed according to the real-time assessment, while 61 (43.8%) passed and 78 (56.1%) failed according to the video assessment. The agreement between the two assessments was 94.9%, with a kappa value of 0.898. Indicating the need for positive pressure ventilation (PPV), 138 (99.3%) did correctly in real-time assessment, while 133 (95.6%) did correctly in video assessment with agreement of 96.4% but low kappa of 0.27. We conclude that the instructors' assessment was excellent and didn't differ from assessments obtained from recorded videos and vice-versa.