Learnable Nonlocal Self-Similarity of Deep Features for Image Denoising

SIAM Journal on Imaging Sciences, Volume 17, Issue 1, Page 441-475, March 2024.
Abstract. High-dimensional deep features extracted by convolutional neural networks have nonlocal self-similarity. However, incorporating this nonlocal prior of deep features into deep network architectures with an interpretable variational framework is rarely explored. In this paper, we propose a learnable nonlocal self-similarity deep feature network for image denoising. Our method is motivated by the fact that the high-dimensional deep features obey a mixture probability distribution based on the Parzen–Rosenblatt window method. Then a regularizer with learnable nonlocal weights is proposed by considering the dual representation of the log-probability prior of the deep features. Specifically, the nonlocal weights are introduced as dual variables that can be learned by unrolling the associated numerical scheme. This leads to nonlocal modules (NLMs) in newly designed networks. Our method provides a statistical and variational interpretation for the nonlocal self-attention mechanism widely used in various networks. By adopting nonoverlapping window and region decomposition techniques, we can significantly reduce the computational complexity of nonlocal self-similarity, thus enabling parallel computation of the NLM. The solution to the proposed variational problem can be formulated as a learnable nonlocal self-similarity network for image denoising. This work offers a novel approach for constructing network structures that consider self-similarity and nonlocality. The improvements achieved by this method are predictable and partially controllable. Compared with several closely related denoising methods, the experimental results show the effectiveness of the proposed method in image denoising.