Layout-your-3D: Controllable and Precise 3D Generation with 2D Blueprint

2024-10-20

arXiv.org

AI Summary

Existing text-to-3D methods struggle to simultaneously ensure physically plausible object interactions and computational efficiency. This paper introduces Layout3D, a controllable and compositional 3D generation framework that leverages user-provided 2D layouts—optionally generated from text—as strong geometric priors. Its key contributions are: (1) the first end-to-end differentiable 3D generation paradigm guided by 2D layouts; (2) a collision-aware global layout optimization coupled with instance-level refinement, jointly ensuring structural physical plausibility and high-fidelity appearance; and (3) an integrated pipeline combining efficient reconstruction initialization, constraint-based optimization, instantiation rendering, and fine-tuning. Experiments demonstrate substantial improvements in geometric合理性 and visual fidelity of generated assets, with per-prompt inference time reduced by multiple orders of magnitude. Moreover, Layout3D natively supports downstream tasks such as 3D editing and object insertion.

Technology Category

Computer Vision: 3D Computer VisionNatural Language Processing: GenerationPlanning, Routing, and Scheduling: Planning with Language Models

Application Category

Economics, Online Markets and Human Computation: Uses of LLMs and GenAI for marketplace design, bidding, and strategic interactionsUser Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendationGraph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

We present Layout-Your-3D, a framework that allows controllable and compositional 3D generation from text prompts. Existing text-to-3D methods often struggle to generate assets with plausible object interactions or require tedious optimization processes. To address these challenges, our approach leverages 2D layouts as a blueprint to facilitate precise and plausible control over 3D generation. Starting with a 2D layout provided by a user or generated from a text description, we first create a coarse 3D scene using a carefully designed initialization process based on efficient reconstruction models. To enforce coherent global 3D layouts and enhance the quality of instance appearances, we propose a collision-aware layout optimization process followed by instance-wise refinement. Experimental results demonstrate that Layout-Your-3D yields more reasonable and visually appealing compositional 3D assets while significantly reducing the time required for each prompt. Additionally, Layout-Your-3D can be easily applicable to downstream tasks, such as 3D editing and object insertion. Our project page is available at:https://colezwhy.github.io/layoutyour3d/

Problem

• Enables controllable 3D generation from text prompts
• Improves plausibility of object interactions in 3D assets
• Reduces optimization time for 3D generation processes

Innovation

• Uses 2D layouts for precise 3D control
• Combines reconstruction models with layout optimization
• Enables fast and coherent 3D asset generation