Generalizability of consensus regarding standardized letters of evaluation competitiveness: A validity study in a national sample of emergency medicine faculty.

IF 1.7 Q2 EDUCATION, SCIENTIFIC DISCIPLINES AEM Education and Training Pub Date : 2024-12-09 eCollection Date: 2024-12-01 DOI:10.1002/aet2.11049

Sharon Bord, Morgan Sehdev, Alexis Pelletier-Bui, Al'ai Alvarez, Benjamin Schnapp, Nicole Dubosh, Caitlin Schrepel, Yoon Soo Park, Eric Shappell

{"title":"Generalizability of consensus regarding standardized letters of evaluation competitiveness: A validity study in a national sample of emergency medicine faculty.","authors":"Sharon Bord, Morgan Sehdev, Alexis Pelletier-Bui, Al'ai Alvarez, Benjamin Schnapp, Nicole Dubosh, Caitlin Schrepel, Yoon Soo Park, Eric Shappell","doi":"10.1002/aet2.11049","DOIUrl":null,"url":null,"abstract":"Background: Standardized letters of evaluation (SLOEs) are an important part of residency recruitment, particularly given the limited availability of other discerning factors in residency applications. While consensus regarding SLOE competitiveness has been studied within a small group of academic faculty, it remains unexplored how a more diverse group of letter readers interpret SLOEs in terms of competitiveness.Methods: A sample of 50 real SLOEs in the new SLOE format (2022 eSLOE 2.0) were selected to match the national rating distribution and anonymized. These SLOEs were ranked in order of competitiveness by 25 faculty members representing diverse demographics, geographic regions, and practice settings. Consensus levels were assessed using previously defined criteria and compared to prior results using a cutoff of ±10% to define a significant difference in consensus levels. Two models were tested to determine their ability to predict consensus rankings: a point-based system and a linear regression model.Results: Faculty consensus in this diverse cohort was slightly below the level measured among academic emergency medicine faculty in the prior study, though no differences were greater than the ±10% cutoff. Prediction models also performed similarly to a previous study except at the tight level of agreement, where consensus was stronger in this study compared to previous results. There is greater consensus among faculty at academic institutions than at community institutions, and years of experience was not correlated with higher consensus.Conclusions: The degree of consensus regarding competitiveness using real SLOEs was similar in this diverse national sample compared to a prior study in a smaller and more homogenous group ranking mock SLOEs. Consensus ranks were predicted with good accuracy using both the point system and the regression model.","PeriodicalId":37032,"journal":{"name":"AEM Education and Training","volume":"8 6","pages":"e11049"},"PeriodicalIF":1.7000,"publicationDate":"2024-12-09","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11628423/pdf/","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"AEM Education and Training","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/10.1002/aet2.11049","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"2024/12/1 0:00:00","PubModel":"eCollection","JCR":"Q2","JCRName":"EDUCATION, SCIENTIFIC DISCIPLINES","Score":null,"Total":0}

引用次数: 0

Abstract

Background: Standardized letters of evaluation (SLOEs) are an important part of residency recruitment, particularly given the limited availability of other discerning factors in residency applications. While consensus regarding SLOE competitiveness has been studied within a small group of academic faculty, it remains unexplored how a more diverse group of letter readers interpret SLOEs in terms of competitiveness.

Methods: A sample of 50 real SLOEs in the new SLOE format (2022 eSLOE 2.0) were selected to match the national rating distribution and anonymized. These SLOEs were ranked in order of competitiveness by 25 faculty members representing diverse demographics, geographic regions, and practice settings. Consensus levels were assessed using previously defined criteria and compared to prior results using a cutoff of ±10% to define a significant difference in consensus levels. Two models were tested to determine their ability to predict consensus rankings: a point-based system and a linear regression model.

Results: Faculty consensus in this diverse cohort was slightly below the level measured among academic emergency medicine faculty in the prior study, though no differences were greater than the ±10% cutoff. Prediction models also performed similarly to a previous study except at the tight level of agreement, where consensus was stronger in this study compared to previous results. There is greater consensus among faculty at academic institutions than at community institutions, and years of experience was not correlated with higher consensus.

Conclusions: The degree of consensus regarding competitiveness using real SLOEs was similar in this diverse national sample compared to a prior study in a smaller and more homogenous group ranking mock SLOEs. Consensus ranks were predicted with good accuracy using both the point system and the regression model.

查看原文