A spatio-temporal inpainting method for Kinect depth video

Abstract

In this paper, we propose a spatio-temporal inpainting method to recover depth images generated by Kinect. Based on the assumption that neighbouring pixels similar in color are likely to have similar depth values, for the first depth image and the sub-images including the motion bodies extracted from the following depth frames, we use color segmentation maps of the corresponding color video frames to guide the depth filling. Considering that some dark regions without valid depth value could lead to the fail of color-segmentation based depth filling, we design a dark region detection method and further refine hole-filling of the unfilled regions with the valid depth values of the same dark region. For the static areas of Kinect depth video, the recovered depth at the same position of previous frame is used to recover the lost depth in the current depth frame. Experimental results show that the proposed method significantly improves depth quality by successfully filling the holes so that we can use it for better 3D rendering.