Score-based Image-to-Image Regression with Synchronized Diffusion

Abstract

Image-to-image regression is an important computer vision task. In this paper, we propose a novel image-to-image regression model following the recent trend in generative modeling that employs Stochastic Differential Equations (SDEs) and score matching. We first apply diffusion processes to regression data using designed SDEs, and then perform inference by gradually reversing the processes. In particular, our method uses synchronized diffusion, which simultaneously applies diffusion to both input and response images to stabilize diffusion and subsequent parameter learning. Furthermore, based on the Expectation-Maximization (EM) algorithm, we develop an effective algorithm for prediction. We implement a conditional U-Net architecture with pre-trained DenseNet encoder for our proposed model and refer to it as DenseSocre. Our new model is able to generate diverse outcomes for image colorization, and the proposed prediction algorithm is able to achieve close to state-of-art performance on high-resolution monocular depth estimation.