SceneConductor: 3D Scene Generation from Single Image with Multi-Agent Orchestration

2026-06-06

AI Summary

This work addresses the challenges of single-image 3D scene generation—namely geometric ambiguity, modeling object relationships, and contextual inference—which are exacerbated by the monolithic architectures and heavy reliance on strong supervision in existing methods, limiting their generalization. To overcome these limitations, the authors propose a multi-agent collaborative framework that decouples the generation process into three stages: scene initialization, environment construction, and multi-agent optimization, enabling efficient and precise modeling through structured decomposition. Key innovations include a novel multi-agent mechanism that jointly ensures local refinement and global consistency, and a geometry-aware layout predictor requiring only segmentation-level annotations, substantially reducing dependence on scene-level supervision. Experiments demonstrate that the proposed method significantly outperforms current state-of-the-art approaches across multiple benchmarks, achieving leading performance in geometric accuracy, spatial consistency, and visual realism.

Abstract

Generating complete 3D scenes from a single image requires inferring globally consistent geometry, object relationships, and environmental context from inherently ambiguous visual evidence. Despite recent progress in joint layout-and-mesh generation, existing methods often rely on holistic or weakly decomposed pipelines that entangle many factors at once and demand extensive scene-level supervision, limiting their generalization to complex real-world environments. We propose a multi-agent orchestration framework that decomposes single-image 3D scene generation into three structured stages: scene initialization, environment construction, and multi-agent refinement. The initialization stage extracts image-derived object masks, builds object-level 3D representations, and predicts an initial spatial layout to form a coarse 3D scene. The environment-construction stage then leverages this initialization together with point-map geometry to build an environmental scaffold of supporting surfaces, room boundaries, materials, and illumination. Finally, in the refinement stage, a planner agent identifies structural and visual inconsistencies, applies simple corrections directly, and dispatches specialist agents for complex localized revisions that are reintegrated into the global scene. To provide reliable structural initialization while reducing reliance on scene-level annotations, we further introduce a geometry-aware layout predictor supervised by sparse geometric priors derived from point maps. Unlike fully supervised layout generators, the predictor can be trained from segmentation-level data and generalizes robustly to diverse real-world scenes. Extensive experiments on benchmark datasets show that our method consistently outperforms prior approaches in geometric accuracy, spatial consistency, and perceptual realism.

Problem

• 3D scene generation
• single-image reconstruction
• scene layout
• geometric consistency
• environmental context

Innovation

• multi-agent orchestration
• 3D scene generation
• geometry-aware layout prediction
• single-image reconstruction
• environmental scaffold