Perceptual visual security index: Analyzing image content leakage for vision language models

Abstract

During the training phase of vision language models (VLMs), the privacy storage and sharing of images are of paramount importance. While the Visual Security Index (VSI) is commonly used for content leakage analysis, it usually focuses on comparing content similarity between plain and protected or encrypted images, neglecting the threat model of visual security. In this paper, considering the functionality of the human visual capability, we comprehensively analyze the system model of VSIs and propose a novel perceptual visual security index (PVSI) to evaluate the content leakage of perceptually encrypted images for VLMs. In particular, we take visual perception (VP) as the adversary’s capability and present the definition of VSI under an honest-but-curious threat model. To evaluate the content leakage of encrypted images under the VP assumption, we first present a robust feature descriptor and obtain the semantic content sets of both plain and encrypted images. Then, we propose a systematic method to reduce the impact of different encryption algorithms. We further evaluate the similarity between semantic content sets to obtain the proposed PVSI. We also analyze the consistency between the proposed visual security definition and PVSI. Extensive experiments are performed on five publicly available image databases. Our experimental results demonstrate that compared with many existing state-of-the-art visual security metrics, the proposed PVSI exhibits better performance not only on images generated from specific image encryption algorithms but also on publicly available image databases.