Elicitation and Evaluation of Hand-based Interaction Language for 3D Conceptual Design in Mixed Reality

Abstract

Conceptual design is a fundamental stage in a design process. Traditional conceptual design tools impose limitations on the intuitive creation process of 3D objects, hindering their full potential. Applying hand-based interaction in mixed reality (MR) provides an immersive and intuitive creation mode without dimensionality transformations, facilitating conceiving ideas in 3D conceptual design. Although studies on hand-based languages have explored different gesture design methodologies and techniques from many individual perspectives, they lack holistic understanding and comprehensive suggestions for gesture application in conceptual design scenarios. This study aims to establish a set of hand-based languages that are available in cognitive, physical, and system-based aspects for 3D conceptual designs in the MR environment. A two-stage study was conducted in an MR environment, combining an elicitation design and a comprehensive evaluation experiment. Through the elicitation design, approximately 930 gesture actions, focusing on 31 targeted functionalities, were collected. By performing an evaluation experiment, a set of theoretically optimal hand-based interaction languages for 3D conceptual design was proposed, and the corresponding guidelines for hand-based interactions were clarified. Our results can further expand human-computer interactions in MR environments and inspire software builders to create novel hand-driven interaction modes or tools.