Spatio-Temporal Convolutional Neural Network for Frame Rate Up-Conversion

Abstract

The visual quality of the video is improved by realizing higher resolution and higher frame rate. In order to realize higher frame rate, we propose new frame rate up-conversion method using spatio-temporal convolutional neural network. In recent years, with the development of machine learning techniques such as convolutional neural networks, clearer interpolation frame estimation has been realized. However, with the conventional convolutional neural network method, it is difficult to estimate an accurate interpolation frames for video including complex motion. In order to deal with this problem, we adopted spatio-temporal convolution rather than conventional spatial convolution. Spatio-temporal convolution is thought to be effective for nonlinear motion because it can capture the time change of the motion of the object. We verified the effectiveness of the proposed method by using video data including complex motions such as rotational motion and scaling.