End-to-End Video Snapshot Compressive Imaging using Video Transformers

Abstract

This paper presents a novel reconstruction algorithm for video Snapshot Compressive Imaging (SCI). Inspired by recent research works on Transformers and Self-Attention mechanism in computer vision, we propose the first video SCI reconstruction algorithm built upon Transformers to capture long-range spatio-temporal dependencies enabling the deep learning of feature maps. Our approach is based on a Spatiotempo-ral Convolutional Multi-head Attention (ST-ConvMHA) which enable to exploit the spatial and temporal information of the video scenes instead of using fully-connected attention layers. To evaluate the performances of our approach, we train our algorithm on DAVIS2017 dataset and we test the trained models on six benchmark datasets. The obtained results in terms of PSNR, SSIM and especially reconstruction time prove the ability of using our reconstruction approach for real-time applications. We truly believe that our research will motivate future works for more video reconstruction approaches.