Haibo Yang, Yang Chen, Yingwei Pan, Ting Yao, Zhineng Chen, Zuxuan Wu, Yu-Gang Jiang, Tao Mei
{"title":"DreamMesh: Jointly Manipulating and Texturing Triangle Meshes for Text-to-3D Generation","authors":"Haibo Yang, Yang Chen, Yingwei Pan, Ting Yao, Zhineng Chen, Zuxuan Wu, Yu-Gang Jiang, Tao Mei","doi":"arxiv-2409.07454","DOIUrl":null,"url":null,"abstract":"Learning radiance fields (NeRF) with powerful 2D diffusion models has\ngarnered popularity for text-to-3D generation. Nevertheless, the implicit 3D\nrepresentations of NeRF lack explicit modeling of meshes and textures over\nsurfaces, and such surface-undefined way may suffer from the issues, e.g.,\nnoisy surfaces with ambiguous texture details or cross-view inconsistency. To\nalleviate this, we present DreamMesh, a novel text-to-3D architecture that\npivots on well-defined surfaces (triangle meshes) to generate high-fidelity\nexplicit 3D model. Technically, DreamMesh capitalizes on a distinctive\ncoarse-to-fine scheme. In the coarse stage, the mesh is first deformed by\ntext-guided Jacobians and then DreamMesh textures the mesh with an interlaced\nuse of 2D diffusion models in a tuning free manner from multiple viewpoints. In\nthe fine stage, DreamMesh jointly manipulates the mesh and refines the texture\nmap, leading to high-quality triangle meshes with high-fidelity textured\nmaterials. Extensive experiments demonstrate that DreamMesh significantly\noutperforms state-of-the-art text-to-3D methods in faithfully generating 3D\ncontent with richer textual details and enhanced geometry. Our project page is\navailable at https://dreammesh.github.io.","PeriodicalId":501480,"journal":{"name":"arXiv - CS - Multimedia","volume":"11 1","pages":""},"PeriodicalIF":0.0000,"publicationDate":"2024-09-11","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"arXiv - CS - Multimedia","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/arxiv-2409.07454","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"","JCRName":"","Score":null,"Total":0}
引用次数: 0
Abstract
Learning radiance fields (NeRF) with powerful 2D diffusion models has
garnered popularity for text-to-3D generation. Nevertheless, the implicit 3D
representations of NeRF lack explicit modeling of meshes and textures over
surfaces, and such surface-undefined way may suffer from the issues, e.g.,
noisy surfaces with ambiguous texture details or cross-view inconsistency. To
alleviate this, we present DreamMesh, a novel text-to-3D architecture that
pivots on well-defined surfaces (triangle meshes) to generate high-fidelity
explicit 3D model. Technically, DreamMesh capitalizes on a distinctive
coarse-to-fine scheme. In the coarse stage, the mesh is first deformed by
text-guided Jacobians and then DreamMesh textures the mesh with an interlaced
use of 2D diffusion models in a tuning free manner from multiple viewpoints. In
the fine stage, DreamMesh jointly manipulates the mesh and refines the texture
map, leading to high-quality triangle meshes with high-fidelity textured
materials. Extensive experiments demonstrate that DreamMesh significantly
outperforms state-of-the-art text-to-3D methods in faithfully generating 3D
content with richer textual details and enhanced geometry. Our project page is
available at https://dreammesh.github.io.
学习辐射场(NeRF)具有强大的二维扩散模型,在文本到三维的生成中颇受欢迎。然而,NeRF 的隐式 3D 表示缺乏对网格和表面纹理的显式建模,而且这种未定义表面的方式可能会出现一些问题,例如纹理细节模糊或跨视角不一致的嘈杂表面。为了解决这些问题,我们提出了 DreamMesh,这是一种新颖的文本到三维架构,它以定义明确的曲面(三角形网格)为中心,生成高保真的三维模型。从技术上讲,DreamMesh 采用了一种独特的从粗到细的方案。在粗略阶段,首先通过文本引导的雅各布因子对网格进行变形,然后 DreamMesh 从多个视角以自由调整的方式交错使用二维扩散模型对网格进行纹理处理。在精细阶段,DreamMesh 对网格进行联合处理,并完善纹理贴图,从而生成具有高保真纹理材质的高质量三角形网格。大量实验证明,DreamMesh 在忠实生成具有更丰富文本细节和增强几何形状的 3D 内容方面,明显优于最先进的文本到 3D 方法。我们的项目页面位于 https://dreammesh.github.io。