Collaborative Streaming and Super Resolution Adaptation for Mobile Immersive Videos

Abstract

Tile-based streaming and super resolution are two representative technologies adopted to improve bandwidth efficiency of immersive video steaming. The former allows selective download of contents in the user viewport by splitting the video into multiple independently decodable tiles. The latter leverages client-side computation to reconstruct the received video into higher quality using advanced neural network models. In this work, we propose CASE, a collaborated adaptive streaming and enhancement framework for mobile immersive videos, which integrates super resolution with tile-based streaming to optimize user experience with dynamic bandwidth and limited computing capability. To coordinate the video transmission and reconstruction in CASE, we identify and address several key design issues including unified video quality assessment, computation complexity model for super resolution, and buffer analysis considering the interplay between transmission and reconstruction. We further formulate the quality-of-experience (QoE) maximization problem for mobile immersive video streaming and propose a rate adaptation algorithm to make the best decisions for download and for reconstruction based on the Lyapunov optimization theory. Extensive evaluation results validate the superiority of our proposed approach, which presents stable performance with considerable QoE improvement, while enabling trade-off between playback smoothness and video quality.