Human-Centered and AI-driven Generation of 6-DoF Extended Reality

Abstract

In order to unlock the full potential of Extended Reality (XR) and its application to societal sectors such as health (e.g., training) or Industry 5.0 (e.g., remote control of infrastructure) there is a need for very realistic environments to enhance the presence of the user. However, current photo-realistic content generation methods (such as Light Fields) require a massive amount of data transmission (i.e., ultra-high bandwidths) and extreme computational power for displaying. Thus, they are not suited for interactive immersive and realistic applications. In this research, we hypothesize that is possible to generate realistic dynamic 3D environments by means of Deep Generative Networks. The work will consist of two parts: (1) a computer vision system that generates the 3D environment based on 2D images, and (2) a Human-Computer Interaction system (HCI) that predicts Region of Interest (RoI) for efficient 3D rendering, subjective and objective assessment of user perception (by means of presence) to enhance the 3D scene quality. This work aims to gain insights into how well deep generative methods can create realistic and immersive environments. This will significantly help future developments in realistic and immersive XR content creation.