Chinese Anesthesiology Milestones in Resident Evaluation: Reliability, Validity, and Correlation with Objective Examination Scores: a Cross-sectional Study.

Abstract

BACKGROUND Evaluating competency acquisition during residency training is crucial. The Anesthesiology Milestones have been implemented in the United States. The China Consortium of Elite Teaching Hospitals for Residency Education has also developed the Chinese Resident Core Competency Milestone Evaluation System. Despite this, Milestones tailored for anesthesiology have yet to be implemented in China. To address this gap, we have developed Chinese Anesthesiology Milestones. This study aims to assess the reliability and validity of the Chinese Anesthesiology Milestones and their correlation with objective examinations. METHODS In this single-center cross-sectional study, we included anesthesia residents enrolled in the standardized residency training program at our hospital during the academic year 2021 to 2022. The Chinese Anesthesiology Milestones were developed based on the American version of Anesthesiology Milestones 2.0 and the Chinese Resident Core Competency Milestone Evaluation System using the Delphi method. The Delphi panel comprised a diverse group, including education administrators, faculty from teaching hospitals, and anesthesia residents. Five attending anesthesiologists independently assessed the levels achieved by each anesthesia resident based on the Chinese Anesthesiology Milestones. Subsequently, they collaboratively discussed the ratings for each resident until a consensus was reached. The interrater reliability, internal consistency, and construct validity were assessed using Kendall's coefficient, Cronbach's α coefficient/ composite reliability, and average variance extracted, respectively. Higher values indicated better reliability or validity. The correlation between Milestone ratings and objective examination scores, including written examinations and Objective Structured Clinical Examinations, were analyzed using Pearson correlation. RESULTS The Chinese Anesthesiology Milestones encompassed 6 competencies, including professionalism, medical knowledge and technical skills, patient care, interpersonal and communication skills, teaching ability, and life-long learning. Milestone evaluation data were available and analyzed from 66 residents. The Kendall's coefficient of concordance among raters ranged from 0.799 (95% confidence interval [CI], 0.793-0.918) to 0.942 (95% CI, 0.934-0.982). The average variance extracted, composite reliability, and Cronbach's α coefficient ranged from 0.782 to 0.920, 0.935 to 0.980, and 0.916 to 0.978, respectively. Correlations between objective examination scores and related Milestone subcompetencies were as follows: written examinations: r = 0.52 (95% CI, 0.22-0.71), technical skills stations: r = 0.51 (95% CI, 0.21-0.71), the oral test station: r = 0.66 (95% CI, 0.45-0.79), and the standardized patient station: r = 0.61 (95% CI, 0.36-0.76). CONCLUSIONS The Chinese Anesthesiology Milestones demonstrated satisfactory interrater reliability, internal consistency, construct validity, and correlation with objective examination scores within our hospital.