It is essential to explore the effect of high-speed rail on energy efficiency to develop medium and long-term green development plans and realize the “dual-carbon target” in China. We utilize urban nighttime light brightness data to measure the energy efficiency of 281 cities in China from 2006 to 2021. We investigate the relationship between the high-speed rail opening and the energy efficiency of cities using the difference-in-differences model. Further, we examine the temporal dynamics of urban energy efficiency improvement by the opening of high-speed rail. We find that the high-speed rail opening significantly improves urban energy efficiency, in which the long-term effect is more influential than the short-term. In addition, we explore the heterogeneity of geographic location, city level, and city size. The results suggest that the high-speed rail opening has a more significant effect on energy efficiency in the eastern region, first-tier cities, and large cities, providing evidence of the “siphon effect” of HSR. Four robustness tests confirm the reliability of the research: replacing core explanatory variables, PSM-DID, counterfactual test, and instrumental variable method.