Image-Based Rendering using Point Cloud for 2D Video Compression

Abstract

The main idea of this paper is to extract the 3D scene geometry for the observed scene and use it for synthesizing a more precise prediction using Image-Based Rendering (IBR) for motion compensation in a hybrid coding scheme. The proposed method first extracts camera parameters using Structure from Motion (SfM). Then, a Patch-based Multi-View Stereo (PMVS) technique is employed to generate the scene Point-Cloud (PC) only from already decoded key-frames. Since the PC could be really sparse in poorly reconstructed regions, a depth expansion mechanism is also used. This 3D information helps to properly warp textures from the key-frames to the target frame. This IBR-based prediction is then used as an additional reference for motion compensation. In this way, the encoder can choose between the rendered prediction and the regular reference pictures through a rate- distortion optimization. On average, the simulation results show about 2.16% bitrate reduction compared to the reference HEVC implementation, for tested dynamic and static scene video sequences.