IEEE Transactions on Visualization and Computer Graphics最新文献_第2页

Audio2Gestures: Generating Diverse Gestures from Audio Audio2Gestures:从音频生成不同的手势

IF 5.2 1区计算机科学 Q1 COMPUTER SCIENCE, SOFTWARE ENGINEERING

IEEE Transactions on Visualization and Computer Graphics

Pub Date : 2023-01-17 DOI: 10.48550/arXiv.2301.06690

Jing Li, Di Kang, Wenjie Pei, Xuefei Zhe, Ying Zhang, Linchao Bao, Zhenyu He

People may perform diverse gestures affected by various mental and physical factors when speaking the same sentences. This inherent one-to-many relationship makes co-speech gesture generation from audio particularly challenging. Conventional CNNs/RNNs assume one-to-one mapping, and thus tend to predict the average of all possible target motions, easily resulting in plain/boring motions during inference. So we propose to explicitly model the one-to-many audio-to-motion mapping by splitting the cross-modal latent code into shared code and motion-specific code. The shared code is expected to be responsible for the motion component that is more correlated to the audio while the motion-specific code is expected to capture diverse motion information that is more independent of the audio. However, splitting the latent code into two parts poses extra training difficulties. Several crucial training losses/strategies, including relaxed motion loss, bicycle constraint, and diversity loss, are designed to better train the VAE. Experiments on both 3D and 2D motion datasets verify that our method generates more realistic and diverse motions than previous state-of-the-art methods, quantitatively and qualitatively. Besides, our formulation is compatible with discrete cosine transformation (DCT) modeling and other popular backbones (i.e. RNN, Transformer). As for motion losses and quantitative motion evaluation, we find structured losses/metrics (e.g. STFT) that consider temporal and/or spatial context complement the most commonly used point-wise losses (e.g. PCK), resulting in better motion dynamics and more nuanced motion details. Finally, we demonstrate that our method can be readily used to generate motion sequences with user-specified motion clips on the timeline.

人们在说同一句话时，可能会受到各种心理和身体因素的影响，做出不同的手势。这种固有的一对多关系使得从音频生成共同语音手势特别具有挑战性。传统的CNN/RNN假设一对一映射，因此倾向于预测所有可能的目标运动的平均值，在推理过程中很容易导致平淡/无聊的运动。因此，我们建议通过将跨模态潜在码划分为共享码和运动特定码来显式地对一对多音频到运动映射进行建模。共享代码被期望负责与音频更相关的运动分量，而运动专用代码被期望捕获更独立于音频的不同运动信息。然而，将潜在代码分为两部分会带来额外的训练困难。几个关键的训练损失/策略，包括放松运动损失、自行车约束和多样性损失，旨在更好地训练VAE。在3D和2D运动数据集上的实验验证了我们的方法在数量和质量上都比以前最先进的方法产生了更真实和多样化的运动。此外，我们的公式与离散余弦变换（DCT）建模和其他流行的主干（即RNN、Transformer）兼容。至于运动损失和定量运动评估，我们发现考虑时间和/或空间上下文的结构化损失/度量（例如STFT）补充了最常用的逐点损失（例如PCK），从而产生更好的运动动力学和更细微的运动细节。最后，我们证明了我们的方法可以很容易地用于在时间线上生成具有用户指定的运动片段的运动序列。

{"title":"Audio2Gestures: Generating Diverse Gestures from Audio","authors":"Jing Li, Di Kang, Wenjie Pei, Xuefei Zhe, Ying Zhang, Linchao Bao, Zhenyu He","doi":"10.48550/arXiv.2301.06690","DOIUrl":"https://doi.org/10.48550/arXiv.2301.06690","url":null,"abstract":"People may perform diverse gestures affected by various mental and physical factors when speaking the same sentences. This inherent one-to-many relationship makes co-speech gesture generation from audio particularly challenging. Conventional CNNs/RNNs assume one-to-one mapping, and thus tend to predict the average of all possible target motions, easily resulting in plain/boring motions during inference. So we propose to explicitly model the one-to-many audio-to-motion mapping by splitting the cross-modal latent code into shared code and motion-specific code. The shared code is expected to be responsible for the motion component that is more correlated to the audio while the motion-specific code is expected to capture diverse motion information that is more independent of the audio. However, splitting the latent code into two parts poses extra training difficulties. Several crucial training losses/strategies, including relaxed motion loss, bicycle constraint, and diversity loss, are designed to better train the VAE. Experiments on both 3D and 2D motion datasets verify that our method generates more realistic and diverse motions than previous state-of-the-art methods, quantitatively and qualitatively. Besides, our formulation is compatible with discrete cosine transformation (DCT) modeling and other popular backbones (i.e. RNN, Transformer). As for motion losses and quantitative motion evaluation, we find structured losses/metrics (e.g. STFT) that consider temporal and/or spatial context complement the most commonly used point-wise losses (e.g. PCK), resulting in better motion dynamics and more nuanced motion details. Finally, we demonstrate that our method can be readily used to generate motion sequences with user-specified motion clips on the timeline.","PeriodicalId":13376,"journal":{"name":"IEEE Transactions on Visualization and Computer Graphics","volume":" ","pages":""},"PeriodicalIF":5.2,"publicationDate":"2023-01-17","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"42570159","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 1

VACSEN: A Visualization Approach for Noise Awareness in Quantum Computing. ：四元计算中Noie Awarens的可视化方法

IF 6.5 1区计算机科学 Q1 COMPUTER SCIENCE, SOFTWARE ENGINEERING

IEEE Transactions on Visualization and Computer Graphics

Pub Date : 2023-01-01 DOI: 10.1109/TVCG.2022.3209455

Shaolun Ruan, Yong Wang, Weiwen Jiang, Ying Mao, Qiang Guan

Quantum computing has attracted considerable public attention due to its exponential speedup over classical computing. Despite its advantages, today's quantum computers intrinsically suffer from noise and are error-prone. To guarantee the high fidelity of the execution result of a quantum algorithm, it is crucial to inform users of the noises of the used quantum computer and the compiled physical circuits. However, an intuitive and systematic way to make users aware of the quantum computing noise is still missing. In this paper, we fill the gap by proposing a novel visualization approach to achieve noise-aware quantum computing. It provides a holistic picture of the noise of quantum computing through multiple interactively coordinated views: a Computer Evolution View with a circuit-like design overviews the temporal evolution of the noises of different quantum computers, a Circuit Filtering View facilitates quick filtering of multiple compiled physical circuits for the same quantum algorithm, and a Circuit Comparison View with a coupled bar chart enables detailed comparison of the filtered compiled circuits. We extensively evaluate the performance of VACSEN through two case studies on quantum algorithms of different scales and in-depth interviews with 12 quantum computing users. The results demonstrate the effectiveness and usability of VACSEN in achieving noise-aware quantum computing.

量子计算由于其比经典计算具有指数级的速度而引起了公众的广泛关注。尽管有这些优势，但今天的量子计算机本质上受到噪音的影响，而且容易出错。为了保证量子算法执行结果的高保真度，告知用户所使用的量子计算机和编译的物理电路的噪声是至关重要的。然而，一种让用户意识到量子计算噪音的直观和系统的方法仍然缺失。在本文中，我们通过提出一种新的可视化方法来实现噪声感知量子计算来填补这一空白。它通过多个交互协调的视图提供了量子计算噪声的整体图像:具有类似电路设计的计算机进化视图概述了不同量子计算机噪声的时间演变，电路滤波视图便于对同一量子算法的多个编译物理电路进行快速滤波，而具有耦合条形图的电路比较视图可以对过滤后的编译电路进行详细比较。我们通过两个不同规模量子算法的案例研究和对12个量子计算用户的深入访谈，广泛评估了VACSEN的性能。结果证明了VACSEN在实现噪声感知量子计算方面的有效性和可用性。

{"title":"VACSEN: A Visualization Approach for Noise Awareness in Quantum Computing.","authors":"Shaolun Ruan, Yong Wang, Weiwen Jiang, Ying Mao, Qiang Guan","doi":"10.1109/TVCG.2022.3209455","DOIUrl":"10.1109/TVCG.2022.3209455","url":null,"abstract":"Quantum computing has attracted considerable public attention due to its exponential speedup over classical computing. Despite its advantages, today's quantum computers intrinsically suffer from noise and are error-prone. To guarantee the high fidelity of the execution result of a quantum algorithm, it is crucial to inform users of the noises of the used quantum computer and the compiled physical circuits. However, an intuitive and systematic way to make users aware of the quantum computing noise is still missing. In this paper, we fill the gap by proposing a novel visualization approach to achieve noise-aware quantum computing. It provides a holistic picture of the noise of quantum computing through multiple interactively coordinated views: a Computer Evolution View with a circuit-like design overviews the temporal evolution of the noises of different quantum computers, a Circuit Filtering View facilitates quick filtering of multiple compiled physical circuits for the same quantum algorithm, and a Circuit Comparison View with a coupled bar chart enables detailed comparison of the filtered compiled circuits. We extensively evaluate the performance of VACSEN through two case studies on quantum algorithms of different scales and in-depth interviews with 12 quantum computing users. The results demonstrate the effectiveness and usability of VACSEN in achieving noise-aware quantum computing.","PeriodicalId":13376,"journal":{"name":"IEEE Transactions on Visualization and Computer Graphics","volume":"29 1","pages":"462-472"},"PeriodicalIF":6.5,"publicationDate":"2023-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"41592083","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

NeRF-Art: Text-Driven Neural Radiance Fields Stylization NeRF-Art:文本驱动的神经辐射领域的风格化

IF 5.2 1区计算机科学 Q1 COMPUTER SCIENCE, SOFTWARE ENGINEERING

IEEE Transactions on Visualization and Computer Graphics

Pub Date : 2022-12-15 DOI: 10.48550/arXiv.2212.08070

Can Wang, Ruixia Jiang, Menglei Chai, Mingming He, Dongdong Chen, Jing Liao

As a powerful representation of 3D scenes, the neural radiance field (NeRF) enables high-quality novel view synthesis from multi-view images. Stylizing NeRF, however, remains challenging, especially in simulating a text-guided style with both the appearance and the geometry altered simultaneously. In this paper, we present NeRF-Art, a text-guided NeRF stylization approach that manipulates the style of a pre-trained NeRF model with a simple text prompt. Unlike previous approaches that either lack sufficient geometry deformations and texture details or require meshes to guide the stylization, our method can shift a 3D scene to the target style characterized by desired geometry and appearance variations without any mesh guidance. This is achieved by introducing a novel global-local contrastive learning strategy, combined with the directional constraint to simultaneously control both the trajectory and the strength of the target style. Moreover, we adopt a weight regularization method to effectively suppress cloudy artifacts and geometry noises which arise easily when the density field is transformed during geometry stylization. Through extensive experiments on various styles, we demonstrate that our method is effective and robust regarding both single-view stylization quality and cross-view consistency. The code and more results can be found on our project page: https://cassiepython.github.io/nerfart/.

作为3D场景的强大表示，神经辐射场（NeRF）能够从多视图图像中合成高质量的新视图。然而，NeRF的样式化仍然具有挑战性，尤其是在模拟外观和几何图形同时更改的文本引导样式时。在本文中，我们介绍了NeRF Art，这是一种文本引导的NeRF风格化方法，通过简单的文本提示来操纵预先训练的NeRF模型的风格。与之前缺乏足够的几何变形和纹理细节或需要网格来指导风格化的方法不同，我们的方法可以在没有任何网格指导的情况下将3D场景转换为以所需几何和外观变化为特征的目标样式。这是通过引入一种新的全局-局部对比学习策略来实现的，该策略结合方向约束来同时控制目标风格的轨迹和强度。此外，我们采用了权重正则化方法来有效地抑制几何风格化过程中密度场变换时容易出现的模糊伪影和几何噪声。通过对各种风格的大量实验，我们证明了我们的方法在单视图风格化质量和跨视图一致性方面是有效和稳健的。代码和更多结果可以在我们的项目页面上找到：https://cassiepython.github.io/nerfart/.

{"title":"NeRF-Art: Text-Driven Neural Radiance Fields Stylization","authors":"Can Wang, Ruixia Jiang, Menglei Chai, Mingming He, Dongdong Chen, Jing Liao","doi":"10.48550/arXiv.2212.08070","DOIUrl":"https://doi.org/10.48550/arXiv.2212.08070","url":null,"abstract":"As a powerful representation of 3D scenes, the neural radiance field (NeRF) enables high-quality novel view synthesis from multi-view images. Stylizing NeRF, however, remains challenging, especially in simulating a text-guided style with both the appearance and the geometry altered simultaneously. In this paper, we present NeRF-Art, a text-guided NeRF stylization approach that manipulates the style of a pre-trained NeRF model with a simple text prompt. Unlike previous approaches that either lack sufficient geometry deformations and texture details or require meshes to guide the stylization, our method can shift a 3D scene to the target style characterized by desired geometry and appearance variations without any mesh guidance. This is achieved by introducing a novel global-local contrastive learning strategy, combined with the directional constraint to simultaneously control both the trajectory and the strength of the target style. Moreover, we adopt a weight regularization method to effectively suppress cloudy artifacts and geometry noises which arise easily when the density field is transformed during geometry stylization. Through extensive experiments on various styles, we demonstrate that our method is effective and robust regarding both single-view stylization quality and cross-view consistency. The code and more results can be found on our project page: https://cassiepython.github.io/nerfart/.","PeriodicalId":13376,"journal":{"name":"IEEE Transactions on Visualization and Computer Graphics","volume":" ","pages":""},"PeriodicalIF":5.2,"publicationDate":"2022-12-15","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"44448914","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 34

What's the Situation with Intelligent Mesh Generation: A Survey and Perspectives 智能网格生成的现状与展望

IF 5.2 1区计算机科学 Q1 COMPUTER SCIENCE, SOFTWARE ENGINEERING

IEEE Transactions on Visualization and Computer Graphics

Pub Date : 2022-11-11 DOI: 10.48550/arXiv.2211.06009

Zezeng Li, Zebin Xu, Ying Li, X. Gu, Na Lei

Intelligent Mesh Generation (IMG) represents a novel and promising field of research, utilizing machine learning techniques to generate meshes. Despite its relative infancy, IMG has significantly broadened the adaptability and practicality of mesh generation techniques, delivering numerous breakthroughs and unveiling potential future pathways. However, a noticeable void exists in the contemporary literature concerning comprehensive surveys of IMG methods. This paper endeavors to fill this gap by providing a systematic and thorough survey of the current IMG landscape. With a focus on 113 preliminary IMG methods, we undertake a meticulous analysis from various angles, encompassing core algorithm techniques and their application scope, agent learning objectives, data types, targeted challenges, as well as advantages and limitations. We have curated and categorized the literature, proposing three unique taxonomies based on key techniques, output mesh unit elements, and relevant input data types. This paper also underscores several promising future research directions and challenges in IMG. To augment reader accessibility, a dedicated IMG project page is available at https://github.com/xzb030/IMG_Survey.

智能网格生成（IMG）是一个新颖而有前途的研究领域，利用机器学习技术生成网格。尽管IMG还处于起步阶段，但它显著拓宽了网格生成技术的适应性和实用性，带来了许多突破，并揭示了未来的潜在途径。然而，在当代文献中，关于IMG方法的全面调查存在着明显的空白。本文试图通过对当前IMG景观进行系统而彻底的调查来填补这一空白。我们重点研究了113种初步的IMG方法，从各个角度进行了细致的分析，包括核心算法技术及其应用范围、代理学习目标、数据类型、有针对性的挑战以及优势和局限性。我们对文献进行了整理和分类，根据关键技术、输出网格单元元素和相关输入数据类型提出了三种独特的分类法。本文还强调了IMG未来几个有前景的研究方向和挑战。为了增加读者的可访问性，IMG项目专用页面可在https://github.com/xzb030/IMG_Survey.

{"title":"What's the Situation with Intelligent Mesh Generation: A Survey and Perspectives","authors":"Zezeng Li, Zebin Xu, Ying Li, X. Gu, Na Lei","doi":"10.48550/arXiv.2211.06009","DOIUrl":"https://doi.org/10.48550/arXiv.2211.06009","url":null,"abstract":"Intelligent Mesh Generation (IMG) represents a novel and promising field of research, utilizing machine learning techniques to generate meshes. Despite its relative infancy, IMG has significantly broadened the adaptability and practicality of mesh generation techniques, delivering numerous breakthroughs and unveiling potential future pathways. However, a noticeable void exists in the contemporary literature concerning comprehensive surveys of IMG methods. This paper endeavors to fill this gap by providing a systematic and thorough survey of the current IMG landscape. With a focus on 113 preliminary IMG methods, we undertake a meticulous analysis from various angles, encompassing core algorithm techniques and their application scope, agent learning objectives, data types, targeted challenges, as well as advantages and limitations. We have curated and categorized the literature, proposing three unique taxonomies based on key techniques, output mesh unit elements, and relevant input data types. This paper also underscores several promising future research directions and challenges in IMG. To augment reader accessibility, a dedicated IMG project page is available at https://github.com/xzb030/IMG_Survey.","PeriodicalId":13376,"journal":{"name":"IEEE Transactions on Visualization and Computer Graphics","volume":" ","pages":""},"PeriodicalIF":5.2,"publicationDate":"2022-11-11","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"41797284","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 1

GPA-Net: No-Reference Point Cloud Quality Assessment with Multi-task Graph Convolutional Network GPA-Net：基于多任务图卷积网络的无参考点云质量评估

IF 5.2 1区计算机科学 Q1 COMPUTER SCIENCE, SOFTWARE ENGINEERING

IEEE Transactions on Visualization and Computer Graphics

Pub Date : 2022-10-29 DOI: 10.48550/arXiv.2210.16478

Ziyu Shan, Qi Yang, Rui Ye, Yujie Zhang, Yi Xu, Xiaozhong Xu, Shan Liu

With the rapid development of 3D vision, point cloud has become an increasingly popular 3D visual media content. Due to the irregular structure, point cloud has posed novel challenges to the related research, such as compression, transmission, rendering and quality assessment. In these latest researches, point cloud quality assessment (PCQA) has attracted wide attention due to its significant role in guiding practical applications, especially in many cases where the reference point cloud is unavailable. However, current no-reference metrics which based on prevalent deep neural network have apparent disadvantages. For example, to adapt to the irregular structure of point cloud, they require preprocessing such as voxelization and projection that introduce extra distortions, and the applied grid-kernel networks, such as Convolutional Neural Networks, fail to extract effective distortion-related features. Besides, they rarely consider the various distortion patterns and the philosophy that PCQA should exhibit shift, scaling, and rotation invariance. In this paper, we propose a novel no-reference PCQA metric named the Graph convolutional PCQA network (GPA-Net). To extract effective features for PCQA, we propose a new graph convolution kernel, i.e., GPAConv, which attentively captures the perturbation of structure and texture. Then, we propose the multi-task framework consisting of one main task (quality regression) and two auxiliary tasks (distortion type and degree predictions). Finally, we propose a coordinate normalization module to stabilize the results of GPAConv under shift, scale and rotation transformations. Experimental results on two independent databases show that GPA-Net achieves the best performance compared to the state-of-the-art no-reference PCQA metrics, even better than some full-reference metrics in some cases. The code is available at: https://github.com/Slowhander/GPA-Net.git.

随着三维视觉的快速发展，点云已经成为越来越受欢迎的三维视觉媒体内容。由于点云结构的不规则性，对压缩、传输、渲染和质量评估等相关研究提出了新的挑战。在这些最新研究中，点云质量评估（PCQA）因其在指导实际应用方面的重要作用而受到广泛关注，尤其是在许多没有参考点云的情况下。然而，目前基于流行的深度神经网络的无参考度量存在明显的缺点。例如，为了适应点云的不规则结构，它们需要预处理，如引入额外失真的体素化和投影，而应用的网格核网络，如卷积神经网络，无法提取有效的失真相关特征。此外，他们很少考虑各种失真模式和PCQA应该表现出移位、缩放和旋转不变性的哲学。在本文中，我们提出了一种新的无参考PCQA度量，称为图卷积PCQA网络（GPA-Net）。为了提取PCQA的有效特征，我们提出了一种新的图卷积核，即GPAConv，它可以专注地捕捉结构和纹理的扰动。然后，我们提出了由一个主任务（质量回归）和两个辅助任务（失真类型和程度预测）组成的多任务框架。最后，我们提出了一个坐标归一化模块来稳定GPAConv在移位、缩放和旋转变换下的结果。在两个独立数据库上的实验结果表明，与最先进的无参考PCQA指标相比，GPA-Net实现了最佳性能，在某些情况下甚至优于一些完全参考指标。该代码位于：https://github.com/Slowhander/GPA-Net.git.

{"title":"GPA-Net: No-Reference Point Cloud Quality Assessment with Multi-task Graph Convolutional Network","authors":"Ziyu Shan, Qi Yang, Rui Ye, Yujie Zhang, Yi Xu, Xiaozhong Xu, Shan Liu","doi":"10.48550/arXiv.2210.16478","DOIUrl":"https://doi.org/10.48550/arXiv.2210.16478","url":null,"abstract":"With the rapid development of 3D vision, point cloud has become an increasingly popular 3D visual media content. Due to the irregular structure, point cloud has posed novel challenges to the related research, such as compression, transmission, rendering and quality assessment. In these latest researches, point cloud quality assessment (PCQA) has attracted wide attention due to its significant role in guiding practical applications, especially in many cases where the reference point cloud is unavailable. However, current no-reference metrics which based on prevalent deep neural network have apparent disadvantages. For example, to adapt to the irregular structure of point cloud, they require preprocessing such as voxelization and projection that introduce extra distortions, and the applied grid-kernel networks, such as Convolutional Neural Networks, fail to extract effective distortion-related features. Besides, they rarely consider the various distortion patterns and the philosophy that PCQA should exhibit shift, scaling, and rotation invariance. In this paper, we propose a novel no-reference PCQA metric named the Graph convolutional PCQA network (GPA-Net). To extract effective features for PCQA, we propose a new graph convolution kernel, i.e., GPAConv, which attentively captures the perturbation of structure and texture. Then, we propose the multi-task framework consisting of one main task (quality regression) and two auxiliary tasks (distortion type and degree predictions). Finally, we propose a coordinate normalization module to stabilize the results of GPAConv under shift, scale and rotation transformations. Experimental results on two independent databases show that GPA-Net achieves the best performance compared to the state-of-the-art no-reference PCQA metrics, even better than some full-reference metrics in some cases. The code is available at: https://github.com/Slowhander/GPA-Net.git.","PeriodicalId":13376,"journal":{"name":"IEEE Transactions on Visualization and Computer Graphics","volume":" ","pages":""},"PeriodicalIF":5.2,"publicationDate":"2022-10-29","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"42738762","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 5

Explore Contextual Information for 3D Scene Graph Generation 探索3D场景图形生成的上下文信息

IF 5.2 1区计算机科学 Q1 COMPUTER SCIENCE, SOFTWARE ENGINEERING

IEEE Transactions on Visualization and Computer Graphics

Pub Date : 2022-10-12 DOI: 10.48550/arXiv.2210.06240

Yu-An Liu, Chengjiang Long, Zhaoxuan Zhang, Bo Liu, Qiang Zhang, Baocai Yin, Xin Yang

3D scene graph generation (SGG) has been of high interest in computer vision. Although the accuracy of 3D SGG on coarse classification and single relation label has been gradually improved, the performance of existing works is still far from being perfect for fine-grained and multi-label situations. In this paper, we propose a framework fully exploring contextual information for the 3D SGG task, which attempts to satisfy the requirements of fine-grained entity class, multiple relation labels, and high accuracy simultaneously. Our proposed approach is composed of a Graph Feature Extraction module and a Graph Contextual Reasoning module, achieving appropriate information-redundancy feature extraction, structured organization, and hierarchical inferring. Our approach achieves superior or competitive performance over previous methods on the 3DSSG dataset, especially on the relationship prediction sub-task.

三维场景图生成(SGG)一直是计算机视觉领域的研究热点。虽然3D SGG在粗分类和单一关系标签上的准确率已经逐步提高，但现有作品在细粒度和多标签情况下的表现还远远不够完美。在本文中，我们为3D SGG任务提出了一个充分挖掘上下文信息的框架，该框架试图同时满足细粒度实体类、多关系标签和高精度的要求。我们提出的方法由图特征提取模块和图上下文推理模块组成，实现了适当的信息冗余特征提取、结构化组织和分层推理。我们的方法在3DSSG数据集上取得了优于或具有竞争力的性能，特别是在关系预测子任务上。

引用次数: 7

Multi-User Redirected Walking in Separate Physical Spaces for Online VR Scenarios 在线VR场景中独立物理空间的多用户重定向行走

IF 5.2 1区计算机科学 Q1 COMPUTER SCIENCE, SOFTWARE ENGINEERING

IEEE Transactions on Visualization and Computer Graphics

Pub Date : 2022-10-07 DOI: 10.48550/arXiv.2210.05356

Sen-Zhe Xu, Jia-Hong Liu, Miao Wang, Fang-Lue Zhang, Songhai Zhang

With the recent rise of Metaverse, online multiplayer VR applications are becoming increasingly prevalent worldwide. However, as multiple users are located in different physical environments, different reset frequencies and timings can lead to serious fairness issues for online collaborative/competitive VR applications. For the fairness of online VR apps/games, an ideal online RDW strategy must make the locomotion opportunities of different users equal, regardless of different physical environment layouts. The existing RDW methods lack the scheme to coordinate multiple users in different PEs, and thus have the issue of triggering too many resets for all the users under the locomotion fairness constraint. We propose a novel multi-user RDW method that is able to significantly reduce the overall reset number and give users a better immersive experience by providing a fair exploration. Our key idea is to first find out the "bottleneck" user that may cause all users to be reset and estimate the time to reset given the users' next targets, and then redirect all the users to favorable poses during that maximized bottleneck time to ensure the subsequent resets can be postponed as much as possible. More particularly, we develop methods to estimate the time of possibly encountering obstacles and the reachable area for a specific pose to enable the prediction of the next reset caused by any user. Our experiments and user study found that our method outperforms existing RDW methods in online VR applications.

随着最近Metaverse的兴起，在线多人虚拟现实应用在全球变得越来越普遍。然而，由于多个用户位于不同的物理环境中，不同的重置频率和时间可能会导致在线协作/竞争VR应用的严重公平性问题。为了保证在线VR应用/游戏的公平性，理想的在线RDW策略必须使不同用户的移动机会均等，无论物理环境布局如何。现有的RDW方法缺乏协调不同pe中多个用户的方案，因此在运动公平性约束下存在触发所有用户过多重置的问题。我们提出了一种新的多用户RDW方法，该方法能够显着减少总体重置次数，并通过提供公平的探索为用户提供更好的沉浸式体验。我们的关键思想是首先找出可能导致所有用户重置的“瓶颈”用户，并根据用户的下一个目标估计重置时间，然后在最大瓶颈时间内将所有用户重定向到有利的姿势，以确保后续重置可以尽可能推迟。更具体地说，我们开发了方法来估计可能遇到障碍物的时间和特定姿势的可到达区域，从而能够预测任何用户引起的下一次重置。我们的实验和用户研究发现，我们的方法在在线VR应用中优于现有的RDW方法。

{"title":"Multi-User Redirected Walking in Separate Physical Spaces for Online VR Scenarios","authors":"Sen-Zhe Xu, Jia-Hong Liu, Miao Wang, Fang-Lue Zhang, Songhai Zhang","doi":"10.48550/arXiv.2210.05356","DOIUrl":"https://doi.org/10.48550/arXiv.2210.05356","url":null,"abstract":"With the recent rise of Metaverse, online multiplayer VR applications are becoming increasingly prevalent worldwide. However, as multiple users are located in different physical environments, different reset frequencies and timings can lead to serious fairness issues for online collaborative/competitive VR applications. For the fairness of online VR apps/games, an ideal online RDW strategy must make the locomotion opportunities of different users equal, regardless of different physical environment layouts. The existing RDW methods lack the scheme to coordinate multiple users in different PEs, and thus have the issue of triggering too many resets for all the users under the locomotion fairness constraint. We propose a novel multi-user RDW method that is able to significantly reduce the overall reset number and give users a better immersive experience by providing a fair exploration. Our key idea is to first find out the \"bottleneck\" user that may cause all users to be reset and estimate the time to reset given the users' next targets, and then redirect all the users to favorable poses during that maximized bottleneck time to ensure the subsequent resets can be postponed as much as possible. More particularly, we develop methods to estimate the time of possibly encountering obstacles and the reachable area for a specific pose to enable the prediction of the next reset caused by any user. Our experiments and user study found that our method outperforms existing RDW methods in online VR applications.","PeriodicalId":13376,"journal":{"name":"IEEE Transactions on Visualization and Computer Graphics","volume":" ","pages":""},"PeriodicalIF":5.2,"publicationDate":"2022-10-07","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"46927440","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 5

TraInterSim: Adaptive and Planning-Aware Hybrid-Driven Traffic Intersection Simulation TraInterSim：自适应和规划感知混合驱动交通交叉口仿真

IF 5.2 1区计算机科学 Q1 COMPUTER SCIENCE, SOFTWARE ENGINEERING

IEEE Transactions on Visualization and Computer Graphics

Pub Date : 2022-10-03 DOI: 10.48550/arXiv.2210.08118

Pei Lv, Xinming Pei, Xinyu Ren, Yuzhen Zhang, Chaochao Li, Mingliang Xu

Traffic intersections are important scenes that can be seen almost everywhere in the traffic system. Currently, most simulation methods perform well at highways and urban traffic networks. In intersection scenarios, the challenge lies in the lack of clearly defined lanes, where agents with various motion plannings converge in the central area from different directions. Traditional model-based methods are difficult to drive agents to move realistically at intersections without enough predefined lanes, while data-driven methods often require a large amount of high-quality input data. Simultaneously, tedious parameter tuning is inevitable involved to obtain the desired simulation results. In this paper, we present a novel adaptive and planning-aware hybrid-driven method (TraInterSim) to simulate traffic intersection scenarios. Our hybrid-driven method combines an optimization-based data-driven scheme with a velocity continuity model. It guides the agent's movements using real-world data and can generate those behaviors not present in the input data. Our optimization method fully considers velocity continuity, desired speed, direction guidance, and planning-aware collision avoidance. Agents can perceive others' motion plannings and relative distances to avoid possible collisions. To preserve the individual flexibility of different agents, the parameters in our method are automatically adjusted during the simulation. TraInterSim can generate realistic behaviors of heterogeneous agents in different traffic intersection scenarios in interactive rates. Through extensive experiments as well as user studies, we validate the effectiveness and rationality of the proposed simulation method.

交通路口是交通系统中几乎随处可见的重要场景。目前，大多数模拟方法在高速公路和城市交通网络中表现良好。在交叉口场景中，挑战在于缺乏明确定义的车道，具有各种运动规划的代理从不同方向聚集在中心区域。传统的基于模型的方法很难在没有足够的预定义车道的情况下驱动代理在十字路口真实地移动，而数据驱动的方法通常需要大量高质量的输入数据。同时，为了获得所需的仿真结果，不可避免地需要进行繁琐的参数调整。在本文中，我们提出了一种新的自适应和规划感知混合驱动方法（TraInterSim）来模拟交通交叉口场景。我们的混合驱动方法将基于优化的数据驱动方案与速度连续性模型相结合。它使用真实世界的数据指导代理的移动，并可以生成输入数据中不存在的行为。我们的优化方法充分考虑了速度连续性、期望速度、方向引导和计划意识防撞。代理可以感知他人的运动计划和相对距离，以避免可能的碰撞。为了保持不同代理的个体灵活性，我们的方法中的参数在模拟过程中会自动调整。TraInterSim可以在交互速率下生成不同交通路口场景下异构代理的真实行为。通过大量的实验和用户研究，我们验证了所提出的模拟方法的有效性和合理性。

{"title":"TraInterSim: Adaptive and Planning-Aware Hybrid-Driven Traffic Intersection Simulation","authors":"Pei Lv, Xinming Pei, Xinyu Ren, Yuzhen Zhang, Chaochao Li, Mingliang Xu","doi":"10.48550/arXiv.2210.08118","DOIUrl":"https://doi.org/10.48550/arXiv.2210.08118","url":null,"abstract":"Traffic intersections are important scenes that can be seen almost everywhere in the traffic system. Currently, most simulation methods perform well at highways and urban traffic networks. In intersection scenarios, the challenge lies in the lack of clearly defined lanes, where agents with various motion plannings converge in the central area from different directions. Traditional model-based methods are difficult to drive agents to move realistically at intersections without enough predefined lanes, while data-driven methods often require a large amount of high-quality input data. Simultaneously, tedious parameter tuning is inevitable involved to obtain the desired simulation results. In this paper, we present a novel adaptive and planning-aware hybrid-driven method (TraInterSim) to simulate traffic intersection scenarios. Our hybrid-driven method combines an optimization-based data-driven scheme with a velocity continuity model. It guides the agent's movements using real-world data and can generate those behaviors not present in the input data. Our optimization method fully considers velocity continuity, desired speed, direction guidance, and planning-aware collision avoidance. Agents can perceive others' motion plannings and relative distances to avoid possible collisions. To preserve the individual flexibility of different agents, the parameters in our method are automatically adjusted during the simulation. TraInterSim can generate realistic behaviors of heterogeneous agents in different traffic intersection scenarios in interactive rates. Through extensive experiments as well as user studies, we validate the effectiveness and rationality of the proposed simulation method.","PeriodicalId":13376,"journal":{"name":"IEEE Transactions on Visualization and Computer Graphics","volume":" ","pages":""},"PeriodicalIF":5.2,"publicationDate":"2022-10-03","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"47313608","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

RankFIRST: Visual Analysis for Factor Investment By Ranking Stock Timeseries. RankFIRST：通过排列股票时间序列进行因子投资的可视化分析。

IF 5.2 1区计算机科学 Q1 COMPUTER SCIENCE, SOFTWARE ENGINEERING

IEEE Transactions on Visualization and Computer Graphics

Pub Date : 2022-09-27 DOI: 10.1109/TVCG.2022.3209414

Huijie Guo, Meijun Liu, Bowen Yang, Ye Sun, Huamin Qu, Lei Shi

In the era of quantitative investment, factor-based investing models are widely adopted in the construction of stock portfolios. These models explain the performance of individual stocks by a set of financial factors, e.g., market beta and company size. In industry, open investment platforms allow the online building of factor-based models, yet set a high bar on the engineering expertise of end-users. State-of-the-art visualization systems integrate the whole factor investing pipeline, but do not directly address domain users' core requests on ranking factors and stocks for portfolio construction. The current model lacks explainability, which downgrades its credibility with stock investors. To fill the gap in modeling, ranking, and visualizing stock time series for factor investment, we designed and implemented a visual analytics system, namely RankFIRST. The system offers built-in support for an established factor collection and a cross-sectional regression model viable for human interpretation. A hierarchical slope graph design is introduced according to the desired characteristics of good factors for stock investment. A novel firework chart is also invented extending the well-known candlestick chart for stock time series. We evaluated the system on the full-scale Chinese stock market data in the recent 30 years. Case studies and controlled user evaluation demonstrate the superiority of our system on factor investing, in comparison to both passive investing on stock indices and existing stock market visual analytics tools.

在量化投资时代，基于因子的投资模型被广泛用于构建股票投资组合。这些模型通过一系列金融因子（如市场贝塔系数和公司规模）来解释个股的表现。在工业领域，开放式投资平台允许在线构建基于因子的模型，但对最终用户的工程专业知识提出了很高的要求。最先进的可视化系统集成了整个因子投资管道，但并不能直接满足领域用户在构建投资组合时对因子和股票进行排序的核心要求。当前的模型缺乏可解释性，降低了其在股票投资者中的可信度。为了填补因子投资在股票时间序列建模、排名和可视化方面的空白，我们设计并实现了一个可视化分析系统，即 RankFIRST。该系统内置了对已建立的因子集合和横截面回归模型的支持，适合人工解读。根据股票投资所需的良好因子特征，引入了分层斜率图设计。此外，还发明了一种新颖的烟花图，将著名的蜡烛图扩展到股票时间序列。我们利用最近 30 年中国股市的完整数据对系统进行了评估。案例研究和受控用户评估表明，与股指被动投资和现有股市可视化分析工具相比，我们的系统在因子投资方面更具优势。

{"title":"RankFIRST: Visual Analysis for Factor Investment By Ranking Stock Timeseries.","authors":"Huijie Guo, Meijun Liu, Bowen Yang, Ye Sun, Huamin Qu, Lei Shi","doi":"10.1109/TVCG.2022.3209414","DOIUrl":"10.1109/TVCG.2022.3209414","url":null,"abstract":"In the era of quantitative investment, factor-based investing models are widely adopted in the construction of stock portfolios. These models explain the performance of individual stocks by a set of financial factors, e.g., market beta and company size. In industry, open investment platforms allow the online building of factor-based models, yet set a high bar on the engineering expertise of end-users. State-of-the-art visualization systems integrate the whole factor investing pipeline, but do not directly address domain users' core requests on ranking factors and stocks for portfolio construction. The current model lacks explainability, which downgrades its credibility with stock investors. To fill the gap in modeling, ranking, and visualizing stock time series for factor investment, we designed and implemented a visual analytics system, namely RankFIRST. The system offers built-in support for an established factor collection and a cross-sectional regression model viable for human interpretation. A hierarchical slope graph design is introduced according to the desired characteristics of good factors for stock investment. A novel firework chart is also invented extending the well-known candlestick chart for stock time series. We evaluated the system on the full-scale Chinese stock market data in the recent 30 years. Case studies and controlled user evaluation demonstrate the superiority of our system on factor investing, in comparison to both passive investing on stock indices and existing stock market visual analytics tools.","PeriodicalId":13376,"journal":{"name":"IEEE Transactions on Visualization and Computer Graphics","volume":"PP ","pages":""},"PeriodicalIF":5.2,"publicationDate":"2022-09-27","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"9509274","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

A Collaborative, Interactive and Context-Aware Drawing Agent for Co-Creative Design 一种协作、交互和上下文感知的协同设计绘图代理

IF 5.2 1区计算机科学 Q1 COMPUTER SCIENCE, SOFTWARE ENGINEERING

IEEE Transactions on Visualization and Computer Graphics

Pub Date : 2022-09-26 DOI: 10.48550/arXiv.2209.12588

F. Ibarrola, Tomas Lawton, Kazjon Grace

Recent advances in text-conditioned generative models have provided us with neural networks capable of creating images of astonishing quality, be they realistic, abstract, or even creative. These models have in common that (more or less explicitly) they all aim to produce a high-quality one-off output given certain conditions, and in that they are not well suited for a creative collaboration framework. Drawing on theories from cognitive science that model how professional designers and artists think, we argue how this setting differs from the former and introduce CICADA: a Collaborative, Interactive Context-Aware Drawing Agent. CICADA uses a vector-based synthesis-by-optimisation method to take a partial sketch (such as might be provided by a user) and develop it towards a goal by adding and/or sensibly modifying traces. Given that this topic has been scarcely explored, we also introduce a way to evaluate desired characteristics of a model in this context by means of proposing a diversity measure. CICADA is shown to produce sketches of quality comparable to a human user's, enhanced diversity and most importantly to be able to cope with change by continuing the sketch minding the user's contributions in a flexible manner.

文本条件生成模型的最新进展为我们提供了能够创建质量惊人的图像的神经网络，无论是逼真的、抽象的，还是创造性的。这些模型的共同点是（或多或少明确地），它们都旨在在特定条件下产生高质量的一次性产出，而且它们不太适合创造性的合作框架。根据认知科学中模拟专业设计师和艺术家思考方式的理论，我们讨论了这种设置与前者的区别，并介绍了CICADA：一种协作、交互式上下文感知的绘图代理。CICADA使用基于矢量的优化合成方法来绘制局部草图（例如用户可能提供的草图），并通过添加和/或合理修改轨迹来实现目标。鉴于这一主题很少被探索，我们还介绍了一种方法，通过提出多样性度量来评估模型在这一背景下的期望特征。CICADA被证明能够制作出与人类用户质量相当的草图，增强了多样性，最重要的是能够通过以灵活的方式继续关注用户的贡献来应对变化。

{"title":"A Collaborative, Interactive and Context-Aware Drawing Agent for Co-Creative Design","authors":"F. Ibarrola, Tomas Lawton, Kazjon Grace","doi":"10.48550/arXiv.2209.12588","DOIUrl":"https://doi.org/10.48550/arXiv.2209.12588","url":null,"abstract":"Recent advances in text-conditioned generative models have provided us with neural networks capable of creating images of astonishing quality, be they realistic, abstract, or even creative. These models have in common that (more or less explicitly) they all aim to produce a high-quality one-off output given certain conditions, and in that they are not well suited for a creative collaboration framework. Drawing on theories from cognitive science that model how professional designers and artists think, we argue how this setting differs from the former and introduce CICADA: a Collaborative, Interactive Context-Aware Drawing Agent. CICADA uses a vector-based synthesis-by-optimisation method to take a partial sketch (such as might be provided by a user) and develop it towards a goal by adding and/or sensibly modifying traces. Given that this topic has been scarcely explored, we also introduce a way to evaluate desired characteristics of a model in this context by means of proposing a diversity measure. CICADA is shown to produce sketches of quality comparable to a human user's, enhanced diversity and most importantly to be able to cope with change by continuing the sketch minding the user's contributions in a flexible manner.","PeriodicalId":13376,"journal":{"name":"IEEE Transactions on Visualization and Computer Graphics","volume":" ","pages":""},"PeriodicalIF":5.2,"publicationDate":"2022-09-26","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"46267533","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 4