Prompting semantic priors for image restoration

Abstract

Restoring high-quality clean images from corrupted observations, commonly referred to as image restoration, has been a longstanding challenge in the computer vision community. Existing methods often struggle to recover fine-grained contextual details due to the lack of semantic awareness of the degraded images. To overcome this limitation, we propose a novel prompt-guided semantic-aware image restoration network, termed PSAIR, which can adaptively incorporate and exploit semantic priors of degraded images and reconstruct photographically fine-grained details. Specifically, we exploit the robust degradation filtering and semantic perception capabilities of the segmentation anything model and utilize it to provide non-destructive semantic priors to aid the network’s semantic perception of the degraded images. To absorb the semantic prior, we propose a semantic fusion module that adaptively utilizes the segmentation map to modulate the features of the degraded image thereby facilitating the network to better perceive different semantic regions. Furthermore, considering that the segmentation map does not provide semantic categories, to better facilitate the network’s customized restoration of different semantics we propose a prompt-guided module which dynamically guides the restoration of different semantics via learnable visual prompts. Comprehensive experiments demonstrate that our PSAIR can restore finer contextual details and thus outperforms existing state-of-the-art methods by a large margin in terms of quantitative and qualitative evaluation.