The Xiangshan volcanic basin hosts the largest volcanic-related uranium deposit in China, while the adjacent Yuhuashan volcanic basin contains only one small uranium deposits. To better understand the metallogenic setting of volcanic-related uranium deposit in the study area, ambient noise data was collected from 142 mobile stations with average spacing about 4 km and 2128 phase velocity dispersion curves were extracted by the two-station method. A shallow crustal shear velocity model was obtained by the direct inversion method of surface wave dispersion. As imaged by the velocity model, the shallow (1.0–2.0 km) high-velocity anomalies correspond to the metamorphic rocks exposed in the Xiangshan and Yuhuashan volcanic basins, whereas the low-velocity anomalous zone with NE trending matches well with the distribution range of the Late Cretaceous red beds. With the deepening of the depth, the Xiangshan volcanic basin exhibits medium to high velocities, while the Yuhuashan volcanic basin demonstrates low velocities. These noticeable structural variations are interpreted to the different tectonic-magmatic activities had occurred in two regions. Based on the inferred basement faults and the strike-slip pull-apart process originated from the NE-trending Fuzhou-Yongfeng Fault, we reveal that the contact zone between the basement faults could have contributed to the emplacement of the uranium-producing volcanic basin, while the faults developed within volcanic cover could have provided favorable conditions for the migration and enrichment of ore-forming fluids. Given that the thickness of overlying volcanic rocks in the Yuhuashan volcanic basin is thicker than that in the Xiangshan volcanic basin, it is speculated that the deep part of the Yuhuashan volcanic basin has better uranium mineralization potential.