Apart from the direct health and behavioral influence of the COVID-19 pandemic itself, COVID-19 rumors as an infodemic enormously amplified public anxiety and cause serious outcomes. Although factors influencing such rumors propagation have been widely studied by previous studies, the role of spatial factors (e.g., proximity to the pandemic) on individuals’ response regarding COVID-19 rumors remain largely unexplored. Accordingly, this study, drawing on the stimulus-organism-response (SOR) framework, examined how proximity to the pandemic (stimulus) influences anxiety (organism), which in turn determines rumor beliefs and rumor outcomes (response). Further, the contingent role of social media usage and health self-efficacy were tested. The research model was tested using 1246 samples via an online survey during the COVID-19 pandemic in China. The results indicate that: (1)The proximity closer the public is to the pandemic, the higher their perceived anxiety; (2) Anxiety increases rumor beliefs, which is further positively associated rumor outcomes; (3) When the level of social media usage is high, the relationship between proximity to the pandemic and anxiety is strengthened; (4) When the level of health self-efficacy is high, the effect of anxiety on rumor beliefs is strengthened and the effect of rumor beliefs on rumor outcomes is also strengthened. This study provides a better understanding of the underlying mechanism of the propagation of COVID-19 rumors from a SOR perspective. Additionally, this paper is one of the first that proposes and empirically verifies the contingent role of social media usage and health self-efficacy on the SOR framework. The findings of study can assist the pandemic prevention department in to efficiently manage rumors with the aim of alleviating public anxiety and avoiding negative outcomes cause by rumors.