This study aims at developing a site similarity characterization method suitable for the site-specific data scenario. The site-specific data is generally multivariate, unique, sparse, incomplete, corrupted, and has temporal and spatial variability, briefly denoted as MUSIC-X. Considering the strong power of Bayesian theory in handling uncertainty, the Bayesian inference framework is employed to build the site-specific multivariate distribution model to characterize the site. Then, by combining the site-specific multivariate distribution model and the image structural similarity (SSIM) theory, a site similarity characterization method under site-specific data scenario is proposed. This proposed method was demonstrated by a real site-specific data in Onsøy site in Norway. The results show that (i) the proposed method can obtain the monotonic site similarity indicator with a range of [0, 1], (ii) site similarity can be assessed from three statistical perspectives, namely mean, standard deviation, and correlation, (iii) the proposed method allows for quantifying the uncertainty associated with site similarity characterization, and (iv) spatial correlation of geo-material parameters can be considered. Besides, the link between similarity and engineering characteristics of sites is revealed by a case study about the bearing capacity analysis of the shallow buried footing foundation.