IEEE transactions on visualization and computer graphics最新文献

Developing convincing and realistic virtual human behavior is essential for enhancing user experiences in virtual reality (VR) and augmented reality (AR) settings. This paper introduces a novel task focused on generating long-term behaviors for virtual agents, guided by specific personality traits and contextual elements within 3D environments. We present a comprehensive framework capable of autonomously producing daily activities autoregressively. By modeling the intricate connections between personality characteristics and observable activities, we establish a hierarchical structure of Needs, Task, and Activity levels. Integrating a Behavior Planner and a World State module allows for the dynamic sampling of behaviors using large language models (LLMs), ensuring that generated activities remain relevant and responsive to environmental changes. Extensive experiments validate the effectiveness and adaptability of our approach across diverse scenarios. This research makes a significant contribution to the field by establishing a new paradigm for personalized and context-aware interactions with virtual humans, ultimately enhancing user engagement in immersive applications. Our project website is at: https://behavior.agent-x.cn/.

The development of AR and VR technologies is enhancing users' online shopping experiences in various ways. However, in existing VR shopping applications, shopping contexts merely refer to the products and virtual malls or metaphorical scenes where users select products. This leads to the defect that users can only imagine rather than intuitively feel whether the selected products are suitable for their real usage scenes, resulting in a significant discrepancy between their expectations before and after the purchase. To address this issue, we propose PaRUS, a VR shopping approach that focuses on the context between products and their real usage scenexns. PaRUS begins by rebuilding the virtual scenario of the products' real usage scene through a new semantic scene reconstruction pipeline (manual operation needed), which preserves both the structured scene and textured object models in the scene. Afterwards, intuitive visualization of how the selected products fit the reconstructed virtual scene is provided. We conducted two user studies to evaluate how PaRUS impacts user experience, behavior, and satisfaction with their purchase. The results indicated that PaRUS significantly reduced the perceived performance risk and improved users' trust and expectation with their results of purchase.

Seamless transition between public dialogue and private talks is essential in everyday conversations. Social Virtual Reality (VR) has revolutionized interpersonal communication by creating a sense of closeness over distance through virtual avatars. However, existing social VR platforms are not successful in providing safety and supporting private conversations, thereby hindering self-disclosure and limiting the potential for meaningful experiences. We approach this problem by exploring the factors affecting private conversations in social VR applications, including the usability of different interaction methods and the awareness with respect to the virtual world. We conduct both expert interviews and a controlled experiment with a social VR prototype we realized. We then leverage the outcomes of the two studies to establish a design space that considers diverse dimensions (including privacy levels, social awareness, and modalities), laying the groundwork for more intuitive and meaningful experiences of private conversation in social VR.

The emergence of spatial immersive technologies allows new ways to collaborate remotely. However, they still need to be studied and enhanced in order to improve their effectiveness and usability for collaborators. Remote Physical Collaborative Extended Reality (RPC-XR) consists in solving augmented physical tasks with the help of remote collaborators. This paper presents our RPC-AR system and a user study evaluating this system during a network hardware assembly task. Our system offers verbal and non-verbal interpersonal communication functionalities. Users embody avatars and interact with their remote collaborators thanks to hand, head and eye tracking, and voice. Our system also captures an environment spatially, in real-time and renders it in a shared virtual space. We designed it to be lightweight and to avoid instrumenting collaborative environments and preliminary steps. It performs capture, transmission and remote rendering of real environments in less than 250ms. We ran a cascading user study to compare our system with a commercial 2D video collaborative application. We measured mutual awareness, task load, usability and task performance. We present an adapted Uncanny Valley questionnaire to compare the perception of remote environments between systems. We found that our application resulted in better empathy between collaborators, a higher cognitive load and a lower level of usability, remaining acceptable, to the remote user. We did not observe any significant difference in performance. These results are encouraging, as participants' observations provide insights to further improve the performance and usability of RPC-AR.

The integration of gaze/eye tracking into virtual and augmented reality devices has unlocked new possibilities, offering a novel human-computer interaction (HCI) modality for on-device extended reality (XR). Emerging applications in XR, such as low-effort user authentication, mental health diagnosis, and foveated rendering, demand real-time eye tracking at high frequencies, a capability that current solutions struggle to deliver. To address this challenge, we present EX-Gaze, an event-based real-time eye tracking system designed for on-device extended reality. EX-Gaze achieves a high tracking frequency of 2KHz, providing decent accuracy and low tracking latency. The exceptional tracking frequency of EX-Gaze is achieved through the use of event cameras, cutting-edge, bio-inspired vision hardware that delivers event-stream output at high temporal resolution. We have developed a lightweight tracking framework that enables real-time pupil region localization and tracking on mobile devices. To effectively leverage the sparse nature of event-streams, we introduce the sparse event-patch representation and the corresponding sparse event patches transformer as key components to reduce computational time. Implemented on Jetson Orin Nano, a low-cost, small-sized mobile device with hybrid GPU and CPU components capable of parallel processing of multiple deep neural networks, EX-Gaze maximizes the computation power of Jetson Orin Nano through sophisticated computation scheduling and offloading between GPUs and CPUs. This enables EX-Gaze to achieve real-time tracking at 2KHz without accumulating latency. Evaluation on public datasets demonstrates that EX-Gaze outperforms other event-based eye tracking methods by striking the best balance between accuracy and efficiency on mobile devices. These results highlight EX-Gaze's potential as a groundbreaking technology to support XR applications that require high-frequency and real-time eye tracking. The code is available at https://github.com/Ningreka/EX-Gaze.

Virtual Reality has been used to improve motivation and help in the visualization of complex computing topics. However, few studies directly compared immersive and non-immersive environments. To address this limitation, we developed Abacus, a programming environment that can run in both immersive and non-immersive modes. We conducted a between-subjects study (n=40), with twenty participants assigned to the desktop mode and twenty participants assigned to the VR mode. Participants used a block-based editor to complete two programming tasks: a non-spatial procedural task, and a spatial 3D math task. We found that VR led to higher gains in self-efficacy and that the gain was significant for participants with lower initial levels of experience and spatial skills.

Virtual reality (VR) enables beyond-real interactions (BRI) that transcend physical constraints, offering effective user experiences like extending a hand to grasp distant objects. However, adapting to novel mappings of BRI often reduces performance and the sense of embodiment. To address this, we propose using noisy tendon electrical stimulation (n-TES) to decrease proprioceptive precision. Previous studies have suggested that attenuating proprioceptive precision is crucial for sensory-motor adaptations. Thus, we hypothesize that n-TES, which has been shown to reduce proprioceptive precision and induce visual-dependent perception in VR, can enhance user adaptation to BRI. We conducted a user study using go-go interaction, a BRI technique for interacting with distant objects, to assess the effects of n-TES. Given the individual variability in n-TES response, participants first underwent a proprioceptive precision test to determine the optimal stimulation intensity to lower the proprioceptive precision from 5 levels (σ = 0.25 - 1.25 mA). Reaching tasks using a 2x2 within-participants design evaluated the effects of go-go interaction and n-TES on performance, subjective task load, and embodiment. Results from 24 participants showed that go-go interaction increased reaching time and task load while decreasing the sense of embodiment. Contrary to our hypothesis, n-TES did not significantly mitigate most of these negative effects of go-go interaction, except that perceived agency was higher with n-TES during go-go interaction. The limited effectiveness of n-TES may be due to participants' habituation or sensory adaptation during the tasks. Future research should consider the adaptation process to BRI and investigate different BRI scenarios.

Recent studies utilized virtual reality (VR) as an "empathy machine" to mitigate bias towards various social groups. However, studies addressing bias against physical disabilities remain scarce, with inconsistent results based on VR experience design. Moreover, most studies assumed the universal effects of VR simulation on bias reduction, ignoring the potential moderating effects of individual characteristics. This study investigated how experience design components and individual characteristics moderate VR simulation's effect on changes in bias towards physical disabilities. We designed a VR wheelchair experience, manipulating the situational context (negative, neutral) and whole-body avatar visualization (visible, invisible). Participants' implicit and explicit bias levels were assessed to examine the changes according to VR design components and individual characteristics (gender, preexisting bias level). Results indicated that following the VR intervention, implicit bias was reduced in the group with higher preexisting bias but rather increased in the group with lower preexisting bias. In addition, gender interacted with avatar visualization such that male participants' implicit bias was reduced with invisible avatars but increased with visible avatars. Explicit bias, in contrast, was reduced regardless of conditions, suggesting the potential response bias in self-report measures. These findings underscore the importance of considering the complex interplay between experience design and individual characteristics in understanding VR's efficacy as an empathy-inducing tool. This study provides insights and guidelines for developing more effective VR interventions to alleviate implicit bias towards physical disabilities.

Accurate reconstruction of heterogeneous scenes for high-fidelity rendering in an efficient manner remains a crucial but challenging task in many Virtual Reality and Augmented Reality applications. The recent 3D Gaussian Splatting (3DGS) has shown impressive quality in scene rendering with real-time performance. However, for heterogeneous scenes with many weak-textured regions, the original 3DGS can easily produce numerously wrong floaters with unbalanced reconstruction using redundant 3D Gaussians, which often leads to unsatisfied scene rendering. This paper proposes a novel multi-plane Gaussian Splatting (MPGS), which aims to achieve high-fidelity rendering with compact reconstruction for heterogeneous scenes. The key insight of our MPGS is the introduction of a novel multi-plane Gaussian optimization strategy, which effectively adjusts the Gaussian distribution for both rich-textured and weak-textured regions in heterogeneous scenes. Moreover, we further propose a multi-scale geometric correction mechanism to effectively mitigate degradation of the 3D Gaussian distribution for compact scene reconstruction. Besides, we regularize the Gaussian distributions using normal information extracted from the compact scene learning. Experimental results on public datasets demonstrate that the proposed MPGS achieves much better rendering quality compared to previous methods, while using less storage and offering more efficient rendering. To our best knowledge, MPGS is a new state-of-the-art 3D Gaussian splatting method for compact reconstruction of heterogeneous scenes, enabling high-fidelity rendering in novel view synthesis, especially improving rendering quality for weak-textured regions. The code will be released at https://github.com/wanglids/MPGS.

The integration of immersive Virtual Reality (I-VR) technology in education has emerged as a promising approach for enhancing learning experiences. There is a handful of research done to study the impact of I-VR on learning outcomes, comparison of learning using I-VR and other traditional learning methods, and the impact of values such as haptic sensation, and verbal and non-verbal cues on the learning outcomes. However, there is a dearth of research on understanding how learning is happening from the perspective of the behavior of the learners in the Virtual Reality Learning Environment (VRLE). To address this gap, we developed an Interaction Behavioral Data (IBD) logging mechanism to log all the interaction traces that constitute the behavior of the learners in a Virtual Reality Learning Environment (VRLE). We deployed the IBD logging mechanism in a VRLE used to learn electromagnetic induction concepts and conducted a study with 30 undergraduate computer science students. We extract the learners' actions from the logged data and contextualize them based on the action features such as duration (Long and Short), and frequency of occurrence (First and Repeated occurrence). In this paper, we investigate the actions extracted from logged interaction trace data to understand the behaviors that lead to high and low performance in the VRLE. Using Epistemic Network Analysis (ENA), we identify differences in prominent actions and co-occurring actions between high and low performers. Additionally, we apply Differential Sequence Mining (DSM) to uncover significant action patterns, involving multiple actions, that are differentially frequent between these two groups. Our findings demonstrate that high performers engage in structured, iterative patterns of experimentation and evaluation, while low performers exhibit less focused exploration patterns. The insights gained from ENA and DSM highlight the behavioral variations between high and low performers in the VRLE, providing valuable information for enhancing learning experiences in VRLEs. These insights gained can be further utilized by the VR content developers to o develop adaptive VR learning content by providing personalized scaffolding leading to the enhancement in the learning process via I-VR.