Development and evaluation of a novel knowledge assessment tool for pediatric emergency medicine clerkships

Abstract

Objectives

This study seeks to determine validity evidence for a newly developed multiple-choice examination (MCE) tool to assess retention and application of medical knowledge of students enrolled in a pediatric emergency medicine (PEM) clerkship.

Methods

A team of PEM physicians created a 110-item MCE covering the range of clinical topics in PEM relevant for medical students. The researchers determined examination content using the report of Clerkship Directors in Emergency Medicine and PEM Interest Group of the Society for Academic Emergency Medicine (SAEM). The authors administered the MCE to fourth-year medical students at the end of their PEM rotation from May 2020 to April 2023 at four institutions and then analyzed the examination using four of Messick's five sources of validity evidence: content, response process, internal structure, and relation to other variables.

Results

A total of 158 students took the test. In academic year (AY)20–21, 47 students took the test and scored, on average, 81%. After revision of poor and indeterminate questions, the 111 medical students who took the revised version of the test in AY21–AY23 scored on average 77.3% with a standard deviation of 5.7% with a normal distribution in scores. The revised questions were rated as excellent (10.0%), good (26.4%), fair (34.5%), poor (24.5%), or indeterminate (4.5%) based on test item discrimination. There was a positive correlation between MCE scores and students' clinical evaluations but no correlation between MCE scores and scores that students received on their clinical notes or patient presentations during case conference.

Conclusions

This novel PEM clerkship examination is a reliable test of medical knowledge. Future directions involve evaluating consequences of the MCE and offering the test to medical students in a dedicated PEM rotation at the national level.