Global Graph-Validated Optimization for VLM-based 3D Indoor Scene Generation

2026-08-03

AI Summary

Existing open-vocabulary 3D indoor scene generation methods often suffer from global semantic inconsistency or physical implausibility due to their reliance on local relationship modeling and optimization procedures sensitive to initialization. To address these limitations, this work proposes a graph-based intermediate representation that decouples semantic consistency from physical feasibility. The approach introduces Global Semantic Validation (GSV) and Global Physical Feasibility Search (GPFS), leveraging a structured graph to explicitly encode semantic constraints. A hybrid strategy combining rule-driven graph validation, evolutionary search, and gradient-based optimization is employed, further enhanced through integration with a vision-language model (VLM). This synergy effectively overcomes the pitfalls of local optima. Evaluated on open-vocabulary 3D layout generation, the method achieves state-of-the-art performance, significantly improving both semantic coherence and physical plausibility of generated scenes.

Abstract

We study open-vocabulary 3D indoor layout generation, which synthesizes diverse and physically plausible scenes from unlabeled 3D assets using free-form language instructions. Recent methods leverage large language models (LLMs) and vision-language models (VLMs) to generate structured scenes from text. However, most model inter-asset relations implicitly or rely on local pairwise constraints and local optimization. These formulations are poorly aligned with the global, highly non-convex layout space, often yielding locally plausible yet globally inconsistent or physically infeasible scenes. We address this problem with a graph-based intermediate representation that separates semantic coherence from physical feasibility, together with a hybrid search-and-refinement strategy. First, Global Semantic Verification (GSV) represents scenes as structured graphs and enforces semantic constraints through rule-based verification. This explicit validation removes contradictory configurations and produces a globally consistent semantic scaffold. Second, Global Physical Feasibility Search (GPFS) combines evolutionary search for global exploration with gradient-based refinement for local exploitation. It reduces dependence on VLM-proposed initialization and improves robustness in non-convex and discontinuous feasible spaces. Together, GSV and GPFS move layout generation beyond local relational modeling and initialization-sensitive optimization toward globally consistent reasoning and search. Experiments show that our method achieves state-of-the-art performance in open-vocabulary 3D indoor layout generation, improving both semantic consistency and physical plausibility.

Problem

• 3D indoor scene generation
• global consistency
• physical feasibility
• open-vocabulary
• layout optimization

Innovation

• graph-based representation
• global semantic verification
• physical feasibility search
• evolutionary optimization
• open-vocabulary 3D generation