Generalizability of consensus regarding standardized letters of evaluation competitiveness: A validity study in a national sample of emergency medicine faculty

Background

Standardized letters of evaluation (SLOEs) are an important part of residency recruitment, particularly given the limited availability of other discerning factors in residency applications. While consensus regarding SLOE competitiveness has been studied within a small group of academic faculty, it remains unexplored how a more diverse group of letter readers interpret SLOEs in terms of competitiveness.

Methods

A sample of 50 real SLOEs in the new SLOE format (2022 eSLOE 2.0) were selected to match the national rating distribution and anonymized. These SLOEs were ranked in order of competitiveness by 25 faculty members representing diverse demographics, geographic regions, and practice settings. Consensus levels were assessed using previously defined criteria and compared to prior results using a cutoff of ±10% to define a significant difference in consensus levels. Two models were tested to determine their ability to predict consensus rankings: a point-based system and a linear regression model.

Results

Faculty consensus in this diverse cohort was slightly below the level measured among academic emergency medicine faculty in the prior study, though no differences were greater than the ±10% cutoff. Prediction models also performed similarly to a previous study except at the tight level of agreement, where consensus was stronger in this study compared to previous results. There is greater consensus among faculty at academic institutions than at community institutions, and years of experience was not correlated with higher consensus.

Conclusions

The degree of consensus regarding competitiveness using real SLOEs was similar in this diverse national sample compared to a prior study in a smaller and more homogenous group ranking mock SLOEs. Consensus ranks were predicted with good accuracy using both the point system and the regression model.