IEEE transactions on visualization and computer graphics最新文献

The speech-driven facial animation technology is generally categorized into two main types: 3D and 2D talking face. Both of these have garnered considerable research attention in recent years. However, to our knowledge, the research into 3D talking face has not progressed as deeply as that of 2D talking face, particularly in terms of lip-sync and perceptual mouth movements. The lip-sync necessitates an impeccable synchronization between mouth motion and speech audio. The speech perception derived from the perceptual mouth movements should resemble that of the driving audio. To mind the gap between the two sub-fields, we propose Learn2Talk, a learning framework that enhances 3D talking face network by integrating two key insights from the field of 2D talking face. Firstly, drawing inspiration from the audio-video sync network, we develop a 3D sync-lip expert model for the pursuit of lip-sync between audio and 3D facial motions. Secondly, we utilize a teacher model, carefully chosen from among 2D talking face methods, to guide the training of the audio-to-3D motions regression network, thereby increasing the accuracy of 3D vertex movements. Extensive experiments demonstrate the superiority of our proposed framework over state-of-the-art methods in terms of lip-sync, vertex accuracy and perceptual movements. Finally, we showcase two applications of our framework: audio-visual speech recognition and speech-driven 3D Gaussian Splatting-based avatar animation. The project page of this paper is: https://lkjkjoiuiu.github.io/Learn2Talk/.

Full-body 3D scanning simplifies the acquisition of digital body models. However, current systems are bulky, intricate, and costly, with strict clothing constraints. We propose a pipeline that combines inner body shape inference and parametric model registration for reconstructing the corresponding body model from a single front scan of a clothed body. Three networks modules (Scan2Front-Net, Front2Back-Net, and Inner2Corr-Net) with relatively independent functions are proposed for predicting front inner, back inner, and parametric model reference point clouds, respectively. We consider the back inner point cloud as an axial offset of the front inner point cloud and divide the body into 14 parts. This offset relationship is then learned within the same body parts to reduce the ambiguity of the inference. The predicted front and back inner point clouds are concatenated as inner body point cloud, and then reconstruction is achieved by registering the parametric body model through a point-to-point correspondence between the reference point cloud and the inner body point cloud. Qualitative and quantitative analysis show that the proposed method has significant advantages in terms of body shape completion and reconstruction body model accuracy.

The paper presents a novel approach to visualizing adversarial robustness (called robustness below) of deep neural networks (DNNs). Traditional tests only return a value reflecting a DNN's overall robustness across a fixed number of test samples. Unlike them, we use test samples to train a generative model (GM) and render a DNN's robustness distribution over infinite generated samples within the GM's latent space. The approach extends test samples, enabling users to obtain new test samples to improve feature coverage constantly. Moreover, the distribution provides more information about a DNN's robustness, enabling users to understand a DNN's robustness comprehensively. We propose three methods to resolve the challenges of realizing the approach. Specifically, we (1) map a GM's high-dimensional latent space onto a plane with less information loss for visualization, (2) design a network to predict a DNN's robustness on massive samples to speed up the distribution rendering, and (3) develop a system to supports users to explore the distribution from multiple perspectives. Subjective and objective experiment results prove the usability and effectiveness of the approach.

Mastering the correct use of laboratory equipment is a fundamental skill for undergraduate science students involved in laboratory-based training. However, hands-on laboratory time is often limited, and remote students may struggle as their absence from the physical lab limits their skill development. An air-displacement micropipette was selected for our initial investigation, as accuracy and correct technique are essential in generating reliable assay data. Handling small liquid volumes demands hand dexterity and practice to achieve proficiency. This research assesses the importance of tactile authenticity during training by faithfully replicating the micropipette's key physical and operational characteristics. We developed a custom haptic training approach called 'Smart Pipette' which promotes accurate operation and enhances laboratory dexterity training. A comparative user study with 34 participants evaluated the effectiveness of the Smart Pipette custom haptic device against training with off-the-shelf hardware, specifically the Quest VR hand controller, which was chosen because it is held mid-air similar to a laboratory micropipette. Both training conditions are integrated with the same self-paced virtual simulation displayed on a computer screen, offering clear video instructions and realtime guidance. Results demonstrated that participants trained with the Smart Pipette custom haptic exhibited increased accuracy and precision while making fewer errors than those trained with off-the-shelf hardware. The Smart Pipette and the Quest VR controller had no significant differences in cognitive load and system usability scores. Tactile authentic interaction devices address challenges faced by online learners, while their applicability extends to traditional classrooms, where real-time feedback significantly enhances overall training performance outcomes.

Data visualization designers and clients need to communicate effectively with each other to achieve a successful project. Unlike a personal or solo project, working with a client introduces a layer of complexity to the process. Client and designer might have different ideas about what is an acceptable solution that would satisfy the goals and constraints of the project. Thus, the client-designer relationship is an important part of the design process. To better understand the relationship, we conducted an interview study with 12 data visualization designers. We develop a model of a client-designer project space consisting of three aspects: surfacing project goals, agreeing on resource allocation, and creating a successful design. For each aspect, designer and client have their own mental model of how they envision the project. Disagreements between these models can be resolved by negotiation that brings them closer to alignment. We identified three main negotiation strategies to navigate the project space: 1) expanding the project space to consider more potential options, 2) constraining the project space to narrow in on the boundaries, and 3) shifting the project space to different options. We discuss client-designer collaboration as a negotiated relationship, with opportunities and challenges for each side. We suggest ways to mitigate challenges to avoid friction from developing into conflict.

When analyzing heterogeneous data comprising numerical and categorical attributes, it is common to treat the different data types separately or transform the categorical attributes to numerical ones. The transformation has the advantage of facilitating an integrated multi-variate analysis of all attributes. We propose a novel technique for transforming categorical data into interpretable numerical feature vectors using Large Language Models (LLMs). The LLMs are used to identify the categorical attributes' main characteristics and assign numerical values to these characteristics, thus generating a multi-dimensional feature vector. The transformation can be computed fully automatically, but due to the interpretability of the characteristics, it can also be adjusted intuitively by an end user. We provide a respective interactive tool that aims to validate and possibly improve the AI-generated outputs. Having transformed a categorical attribute, we propose novel methods for ordering and color-coding the categories based on the similarities of the feature vectors.

Nowadays, orthodontics has become an important part of modern personal life to assist one in improving mastication and raising self-esteem. However, the quality of orthodontic treatment still heavily relies on the empirical evaluation of experienced doctors, which lacks quantitative assessment and requires patients to visit clinics frequently for in-person examination. To resolve the aforementioned problem, we propose a novel and practical mobile device-based framework for precisely measuring tooth movement in treatment, so as to simplify and strengthen the traditional tooth monitoring process. To this end, we formulate the tooth movement monitoring task as a multi-view multi-object pose estimation problem via different views that capture multiple texture-less and severely occluded objects (i.e. teeth). Specifically, we exploit a pre-scanned 3D tooth model and a sparse set of multi-view tooth images as inputs for our proposed tooth monitoring framework. After extracting tooth contours and localizing the initial camera pose of each view from the initial configuration, we propose a joint pose estimation scheme to precisely estimate the 3D pose of each individual tooth, so as to infer their relative offsets during treatment. Furthermore, we introduce the metric of Relative Pose Bias to evaluate the individual tooth pose accuracy in a small scale. We demonstrate that our approach is capable of reaching high accuracy and efficiency as practical orthodontic treatment monitoring requires.

Dimensionality reduction (DR) techniques help analysts to understand patterns in high-dimensional spaces. These techniques, often represented by scatter plots, are employed in diverse science domains and facilitate similarity analysis among clusters and data samples. For datasets containing many granularities or when analysis follows the information visualization mantra, hierarchical DR techniques are the most suitable approach since they present major structures beforehand and details on demand. This work presents HUMAP, a novel hierarchical dimensionality reduction technique designed to be flexible on preserving local and global structures and preserve the mental map throughout hierarchical exploration. We provide empirical evidence of our technique's superiority compared with current hierarchical approaches and show a case study applying HUMAP for dataset labelling.

In today's data-rich environment, visualization literacy-the ability to understand and communicate information through charts-is increasingly important. However, constructing effective charts can be challenging due to the numerous design choices involved. Off-the-shelf systems and libraries produce charts with carefully selected defaults that users may not be aware of, making it hard to increase their visualization literacy with those systems. In addition, traditional ways of improving visualization literacy, such as textbooks and tutorials, can be burdensome as they require sifting through a plethora of resources. To address this challenge, we designed Iguanodon, an easy-to-use game application that complements the traditional methods of improving visualization construction literacy. In our game application, users interactively choose whether to apply design choices, which we assign to sub-tasks that must be optimized to create an effective chart. The application offers multiple game variations to help users learn how different design choices should be applied to construct effective charts. Furthermore, our approach easily adapts to different visualization design guidelines. We describe the application's design and present the results of a user study with 37 participants. Our findings indicate that our game-based approach supports users in improving their visualization literacy.