Enhanced 3D Generation by 2D Editing

2024-12-08

arXiv.org

AI Summary

Existing Score Distillation Sampling (SDS)-based 3D generation methods rely on single-step 2D denoising, leading to over-smoothed textures, impoverished geometric details, and limited content diversity. To address these limitations, we propose GE3D—a novel framework that formulates 3D generation as a multi-step latent-space 2D editing process. GE3D introduces a dual-trajectory alignment mechanism that jointly optimizes a noise-preserved fidelity trajectory and a text-guided denoising trajectory, enabling deep coupling between 3D representations and 2D diffusion priors. Through latent-space trajectory alignment, multi-granularity information distillation, and iterative editing refinement, GE3D significantly improves texture fidelity, geometric detail, and multi-view consistency of generated 3D assets. Quantitative and qualitative evaluations demonstrate state-of-the-art performance in material realism and text-3D alignment. The code and demo are publicly available.

Technology Category

Computer Vision: 3D Computer VisionNatural Language Processing: GenerationMachine Learning: Deep Generative Models & Autoencoders

Application Category

Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAGSemantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMsEconomics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications

Abstract

Distilling 3D representations from pretrained 2D diffusion models is essential for 3D creative applications across gaming, film, and interior design. Current SDS-based methods are hindered by inefficient information distillation from diffusion models, which prevents the creation of photorealistic 3D contents. Our research reevaluates the SDS approach by analyzing its fundamental nature as a basic image editing process that commonly results in over-saturation, over-smoothing and lack of rich content due to the poor-quality single-step denoising. To address these limitations, we propose GE3D (3D Generation by Editing). Each iteration of GE3D utilizes a 2D editing framework that combines a noising trajectory to preserve the information of the input image, alongside a text-guided denoising trajectory. We optimize the process by aligning the latents across both trajectories. This approach fully exploits pretrained diffusion models to distill multi-granularity information through multiple denoising steps, resulting in photorealistic 3D outputs. Both theoretical and experimental results confirm the effectiveness of our approach, which not only advances 3D generation technology but also establishes a novel connection between 3D generation and 2D editing. This could potentially inspire further research in the field. Code and demos are released at https://jahnsonblack.github.io/GE3D/.

Problem

• Inefficient distillation from 2D diffusion models to 3D.
• Over-saturation and over-smoothing in current SDS-based methods.
• Lack of rich content and diversity in 3D generation.

Innovation

• GE3D method combines noising and denoising trajectories
• Aligns latents across trajectories for optimized 3D generation
• Utilizes pretrained diffusion models for photorealistic outputs