DreamScene: 3D Gaussian-based End-to-end Text-to-3D Scene Generation

2025-07-18

AI Summary

This work addresses key challenges in text-to-3D scene generation—low automation, poor geometric-semantic consistency, and limited fine-grained editability—by proposing the first end-to-end differentiable framework for text-to-3D scene synthesis and 4D dynamic editing. Methodologically, it integrates GPT-4–driven semantic scene planning, hybrid graph-structured modeling, progressive differentiable camera sampling, and multi-step Gaussian splatting reconstruction, augmented by Formation Pattern Sampling and global consistency optimization. Experiments demonstrate substantial improvements over state-of-the-art methods in generation fidelity, 3D structural coherence, and interactive editing flexibility. The framework enables high-quality, editable open-domain 3D content generation for both indoor and outdoor scenes, exhibiting strong practical deployability.

Technology Category

• Natural Language Processing: Generation
• Computer Vision: 3D Computer Vision
• Humans and AI: Game Design — Procedural Content Generation & Storytelling

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Generating 3D scenes from natural language holds great promise for applications in gaming, film, and design. However, existing methods struggle with automation, 3D consistency, and fine-grained control. We present DreamScene, an end-to-end framework for high-quality and editable 3D scene generation from text or dialogue. DreamScene begins with a scene planning module, where a GPT-4 agent infers object semantics and spatial constraints to construct a hybrid graph. A graph-based placement algorithm then produces a structured, collision-free layout. Based on this layout, Formation Pattern Sampling (FPS) generates object geometry using multi-timestep sampling and reconstructive optimization, enabling fast and realistic synthesis. To ensure global consistent, DreamScene employs a progressive camera sampling strategy tailored to both indoor and outdoor settings. Finally, the system supports fine-grained scene editing, including object movement, appearance changes, and 4D dynamic motion. Experiments demonstrate that DreamScene surpasses prior methods in quality, consistency, and flexibility, offering a practical solution for open-domain 3D content creation. Code and demos are available at https://dreamscene-project.github.io.

Problem

• Automating 3D scene generation from text with consistency
• Achieving fine-grained control in editable 3D scene creation
• Ensuring global consistency in indoor and outdoor 3D scenes

Innovation

• GPT-4 agent constructs hybrid semantic graph
• Graph-based algorithm ensures collision-free layout
• Progressive camera sampling maintains 3D consistency