Scene-SAM3D: Multi-View Scene Asset Generation Without Fine-Tuning

2026-07-18

AI Summary

Existing single-view 3D generation methods struggle to produce high-quality, multi-view-consistent 3D assets in real-world scenarios due to occlusions, redundant observations, and cross-view inconsistencies. This work proposes an efficient, training-free framework that selects complementary views to fuse multi-view information in latent space while suppressing conflicting signals, followed by a lightweight rigid-body Gaussian optimization for rapid layout refinement. For the first time, SAM3D is extended to multi-view scene generation, enabling the construction of high-fidelity 3D scenes without fine-tuning. Evaluated on Replica and ScanNet++, the method reduces scene-level Chamfer Distance by 43.8% and 30.9%, respectively, while cutting sampling FLOPs and runtime latency by nearly 20%.

Abstract

High-quality 3D scene assets are critical for embodied applications such as robotic manipulation, navigation, and simulation. Despite their strong object priors, recent single-image 3D generation models such as SAM3D remain insufficient for real-world scenes, where severe occlusions, redundant observations, and cross-view inconsistencies make reliable scene generation challenging. We introduce Scene-SAM3D, a training-free framework that extends SAM3D from single-view object generation to calibrated multi-view scene asset generation. Scene-SAM3D selects a compact set of complementary views, reducing observation redundancy while providing additional evidence for regions occluded in individual views. Based on the selected views, it performs step-efficient latent velocity fusion to integrate multi-view evidence and suppress cross-view conflicts in canonical space. Finally, a lightweight rigid-object Gaussian optimization refines the scene layout within 200 iterations while preserving the generated object geometry. Experiments on Replica and ScanNet++ demonstrate consistent improvements at both instance and scene levels, with our method reducing scene-level CD by 43.8% on Replica and 30.9% on ScanNet++, while cutting flow-model sampling FLOPs and wall-time latency by nearly 20% under the same multi-view setting. Code will be released at https://github.com/xibi777/Scene-SAM3D.

Problem

• 3D scene generation
• multi-view consistency
• occlusion handling
• scene-level reconstruction
• cross-view conflict

Innovation

• multi-view 3D generation
• training-free framework
• latent velocity fusion
• Gaussian optimization
• scene asset generation