Measuring the Quality of the OSCE in a Chiropractic Programme: A Review of Metrics and Recommendations.

Objectives: The Objective Structured Clinical Examination (OSCE) is a commonly used assessment of clinical skill, and ensuring the quality and reliability in OSCEs is a complex, and on-going process. This paper discusses scoring schemas and reviews checklists and global rating scales (GRS) for marking. Also detailed are post-examination quality assurance metrics tailored to smaller cohorts, with an illustrative dataset.

Methods: A deidentified OSCE dataset, from stations with both a checklist and GRS, of 24 examinees from a 2021 cohort was assessed using the following metrics: Cut-scores or pass-rates, number-of-failures, R2, inter-grade discrimination, and between-group-variation. The results were used to inform a set of implementable recommendations to improve future OSCEs.

Results: For most stations the calculated cut-score calculated was higher than the traditional pass of 50% (58.9.8-68.4%). Number-of-failures were low for traditional pass rates and cuts-scores (0.00-16.7%), excepting Lab Analysis where number-of-failures was 50.0%. R2 values ranged from 0.67-0.97, but proportion of total variance was high (67.3-95.9). These data suggest there were potential missed teaching concepts, that station marking was open to examiner interpretation, and there were inconsistencies in examiner marking.Recommendations included increasing checklist detail and using a weighted marking scale, separating some stations into dichotomous and key-feature checklists, using GRSs specific to each station, and reviewing all future OSCEs with the metrics described to guide refinements.

Conclusions: The analysis used revealed several potential issues with the OSCE assessment. These findings informed recommendations to improve the quality of future examinations.