Using Segmentation to Enhance Frame Prediction in a Multi-Scale Spatial-Temporal Feature Extraction Network

Abstract

Designing a machine to predict future events is a challenging problem to even existing state-of-the-art approaches. It require great computation power either in adversarial training and in segmentation and optical flow. By combining conventional segmentation and the DNN we proposed in this paper, we have a simpler architecture which effectively and efficiently predicts both future frames and semantics more precise than the previous approaches. The input is a raw image sequence, and each frame of it is segmented for semantics, extracted for spatial features, analyzed for temporal features at different scales in a top down path; and then the prediction of frames and segmentation are synthesized in the bottom-up path. Results of our model show superiority of prediction to other state-of-the-art ones in (1) precision of frames, and (2) accuracy of segmentation masks.