Coordinated 3D interaction in tablet- and HMD-based hybrid virtual environments

Traditional 3D User Interfaces (3DUI) in immersive virtual reality can be inefficient in tasks that involve diversities in scale, perspective, reference frame, and dimension. This paper proposes a solution to this problem using a coordinated, tablet- and HMD-based, hybrid virtual environment system. Wearing a non-occlusive HMD, the user is able to view and interact with a tablet mounted on the non-dominant forearm, which provides a multi-touch interaction surface, as well as an exocentric God view of the virtual world. To reduce transition gaps across 3D interaction tasks and interfaces, four coordination mechanisms are proposed, two of which were implemented, and one was evaluated in a user study featuring complex level-editing tasks. Based on subjective ratings, task performance, interview feedback, and video analysis, we found that having multiple Interaction Contexts (ICs) with complementary benefits can lead to good performance and user experience, despite the complexity of learning and using the hybrid system. The results also suggest keeping 3DUI tasks synchronized across the ICs, as this can help users understand their relationships, smoothen within- and between-task IC transitions, and inspire more creative use of different interfaces.