Hierarchical coarse to fine depth estimation for realistic view interpolation

Abstract

This paper presents a novel approach for view synthesis and image interpolation. The algorithm is build up in a hierarchical way, and this on different structural levels instead of using a classic image pyramid. First coarse matching is done on a 'shape basis' only. A background-foreground segmentation yields a fairly accurate contour for every incoming video stream. Inter-relating these contours is a 1D problem and as such very fast. This step is then used to compute small position dependent bounding-boxes in 3D space which enclose the underlying object. The next step is a more expensive window based matching, within the volume of these bounding-boxes. This is limited to a number of regions around 'promising' feature points. Global regularisation is obtained by a graph cut. Speed results here from limiting the number of feature points. In a third step the interpolation is 'pre-rendered' and simultaneously evaluated on a per pixel basis. This is done by computing a Birchfield dissimilarity measure on the GPU. Per pixel parallelised operations keep computational cost low. Finally the bad interpolated parts are 'patched'. This per pixel correction yields the final interpolated view at the finest level. Here we also deal explicitly with opacity at the borders of the foreground object.