Assessing competency in less invasive surfactant administration: simulation-based validity evidence for the LISA-AT scores.

Abstract

Background: The Less Invasive Surfactant Administration Assessment Tool (LISA-AT) was developed to support operator training and competence assessment. This study aimed to gather validity evidence in the simulated setting to support using the LISA-AT scores.

Methods: Validity evidence was gathered using the Messick framework. The lowest quartile (Q1) for the median of the experts' LISA-AT scores defined the minimum passing score.

Results: Ten experts and 23 novices were enrolled in this study. Eight of the original 15 LISA-AT metrics effectively discriminated between novices and experts and demonstrated high test-retest reliability (Spearman's rho = 0.87), high internal consistency, and good inter-rater reliability (Cronbach's alpha = 0.88 and 0.82, respectively). The LISA-AT discriminated between novices' and experts' first two attempts with median [IQR] scores of 29 [26-32] vs 39 [39-40]). The minimum passing score was defined as 39/40 points, and the novices used a median [IQR] of 6 [5-7] attempts, ranging from 4 to 9 attempts, to reach this score. Compared with the expert group, the novices' laryngoscopy skills and time remained significantly different even after attaining the minimum passing score.

Conclusion: We found strong validity evidence to support using the LISA-AT scores to train new LISA operators to the minimum passing score to ensure competence in the simulated setting.

Impact: This study showed robust validity evidence for using the Less Invasive Surfactant Administration Assessment Tool (LISA-AT) to train LISA novices with a high test-retest reliability, high internal consistency, and good inter-rater reliability. Eight of the original 15 LISA-AT metrics effectively discriminated between novices and experts, with a minimum passing score of 39/40 points, corresponding to the lowest quartile (Q1) of the experts' performances. Novices typically needed six attempts in the simulated setting, ranging from four to nine. However, their laryngoscopy skills and duration remained significantly different from experts. Using the LISA-AT score can ensure competence in the simulated setting before advancing to supervised clinical procedures.