Sentiment analysis of letters of recommendation for a U.S. pain medicine fellowship from 2020 to 2023.

Abstract

Objectives: Letters of recommendation (LORs) are an important part of pain medicine fellowship applications that may be subject to implicit bias by the letter's author. This study evaluated letters of recommendation for applications to pain medicine fellowships in the United States to characterize biases and differences among applicants over four application cycles.

Methods: This was a retrospective single-site cohort study. De-identified LORs were collected from 2020 to 2023 from one institution. The Valence Aware Dictionary and sEntiment Reasoner (VADER) natural language processing package scored positive LOR sentiment. In addition, the deep learning tool, Empath, scored LORs for 15 sentiments. Wilcoxon rank-sum and one-way ANOVA tests compared scores between applicant demographics: gender, race, medical school type, residency specialty, and chief resident status, as well as letter writers' academic position.

Results: Nine hundred and sixty-four applications were studied over four application cycles. Program directors wrote fewer words (p = 0.020) and less positively (p < 0.001) compared to department chairs and letter writers with neither position. Department chairs wrote with less "negative emotion" compared to both program directors and writers with neither position (p < 0.001). Anesthesiologist applicants received more letters highlighting "achievement" (p < 0.001) while PM&R applicants submitted letters with less "negative emotion" (p < 0.001) compared to other specialties. Chief residents' letters scored higher in "leader" sentiment (p < 0.001) and lower in "negative emotion" (p < 0.001).

Discussion: Linguistic content did not favor certain genders or races over others. However, disparities in LORs do exist depending on an applicant's specialty and chief resident status, as well as the academic status of the letter writer.