The COVID-19 pandemic had a profound effect on health workers; one of the most affected occupational groups. In Sierra Leone, health workers were already scarred by the 2014 Ebola outbreak that claimed many lives; this study aimed to explore the experiences of health workers during the COVID-19 pandemic, with particular attention to this context. In-depth interviews were conducted with 24 health workers from three regions in Sierra Leone. The interviews were recorded, transcribed, and analyzed using thematic analysis. The study revealed that the COVID-19 pandemic had a substantial negative impact on the emotional and mental well-being of health workers, with exposure risk being the primary source of stress. The experiences of the 2014 Ebola outbreak shaped their perceptions of COVID-19 while stigma from the community and social circle was perceived as discriminatory and stressful. As the COVID-19 outbreak unfolded, health workers noticed a lower mortality rate relative to their expectations and this strengthened their conviction that the outbreak was less severe than the Ebola outbreak. Health workers believe that lessons from the Ebola outbreak were instrumental in managing COVID-19 effectively. The findings highlight the critical importance of supporting the emotional and mental health of health workers during infectious disease outbreaks, especially in low-income settings where mental health resources are sparse. Preparedness for future disease outbreaks should incorporate lessons learned from site-specific previous outbreaks, i.e., in Sierra Leone, from both the COVID-19 and Ebola, and establishing dedicated counseling and emotional support programs for health workers during crises.