Hydrogen is an important energy source for achieving energy transition and carbon neutrality. Underground salt caverns are ideal for large-scale and long-term underground hydrogen storage (UHS). The hydrogen diffusivity in the rock of a salt cavern gas storage is an important parameter for evaluating the hydrogen tightness of UHS. In this study, hydrogen diffusivity was investigated and the effects of the rock microstructure in a UHS on hydrogen diffusivity were analyzed by using experiments and numerical modeling. Core samples were drilled from the Chuzhou Salt Cavern Gas Storage in the Huai'an salt mine and tested for hydrogen diffusivity using a newly developed setup. A core hydrogen diffusion coefficient model was constructed using the time-domain diffusion method to examine the effects of the complex microstructure and mineral distribution of salt rocks and mudstones. The results indicated the following: (1) the diffusion coefficient of the interlayer mudstone were approximately two orders of magnitude larger than those of salt rocks, and the mudstone showed obvious bidisperse diffusion behavior; (2) high temperature and pressure facilitated hydrogen diffusivity in the cores; (3) the diffusion coefficient of mineral particles was of the order of 10−11 m2/s, which was significantly lower than that of core samples; (4) microscale modeling of hydrogen diffusion using the digital rock method revealed that the discrete microfractures and pores in the cores contributed considerably to the hydrogen diffusion process in tight rocks.