Part-aware Shape Generation with Latent 3D Diffusion of Neural Voxel Fields

2024-05-02

arXiv.org

AI Summary

This work addresses part-aware 3D shape generation, tackling the longstanding challenge of jointly achieving high geometric fidelity, fine-grained texture detail, and structural consistency across semantic parts. We propose an implicit 3D diffusion-driven neural voxel field framework: (1) for the first time, we apply diffusion modeling directly in the parameter space of implicit voxel fields—bypassing memory-intensive explicit voxel grids; (2) we design a part-aware decoder that explicitly models and co-generates semantic parts via semantics-guided part embedding encoding; and (3) we integrate differentiable voxel rendering for end-to-end optimization. Evaluated on four object categories, our method significantly outperforms state-of-the-art approaches, yielding generated shapes with superior geometric accuracy, precise part segmentation, and higher visual quality in rendered outputs.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Deep Generative Models & Autoencoders
• Natural Language Processing: Generation

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation

Abstract

This paper presents a novel latent 3D diffusion model for the generation of neural voxel fields, aiming to achieve accurate part-aware structures. Compared to existing methods, there are two key designs to ensure high-quality and accurate part-aware generation. On one hand, we introduce a latent 3D diffusion process for neural voxel fields, enabling generation at significantly higher resolutions that can accurately capture rich textural and geometric details. On the other hand, a part-aware shape decoder is introduced to integrate the part codes into the neural voxel fields, guiding the accurate part decomposition and producing high-quality rendering results. Through extensive experimentation and comparisons with state-of-the-art methods, we evaluate our approach across four different classes of data. The results demonstrate the superior generative capabilities of our proposed method in part-aware shape generation, outperforming existing state-of-the-art methods.

Problem

• Generate part-aware 3D shapes with neural voxel fields
• Enable high-resolution 3D diffusion for detailed geometry
• Improve part decomposition accuracy in shape generation

Innovation

• Latent 3D diffusion for neural voxel fields
• Part-aware shape decoder integration
• High-resolution geometric detail capture