HiScene: Creating Hierarchical 3D Scenes with Isometric View Generation

2025-04-17

AI Summary

To address weak category generalization and limited interactive editability in scene-level 3D generation, this paper proposes a hierarchical diffusion-based framework that models rooms as decomposable “composite objects” and jointly optimizes 2D semantic layouts and 3D geometry within an isometric projection. Our key contributions are: (1) the first scene–room–object three-tier generative paradigm; (2) a modality-agnostic video diffusion mechanism for geometric completion, effectively mitigating occlusion- and shadow-induced structural incompleteness; and (3) integration of shape priors to enforce spatial consistency and physical plausibility. Experiments demonstrate substantial improvements in layout naturalness, alignment with user intent, and support for interactive editing. The method achieves state-of-the-art performance on open-category scene generation benchmarks.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Natural Language Processing: Generation
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Uses of LLMs and GenAI for marketplace design, bidding, and strategic interactions
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems

Abstract

Scene-level 3D generation represents a critical frontier in multimedia and computer graphics, yet existing approaches either suffer from limited object categories or lack editing flexibility for interactive applications. In this paper, we present HiScene, a novel hierarchical framework that bridges the gap between 2D image generation and 3D object generation and delivers high-fidelity scenes with compositional identities and aesthetic scene content. Our key insight is treating scenes as hierarchical"objects"under isometric views, where a room functions as a complex object that can be further decomposed into manipulatable items. This hierarchical approach enables us to generate 3D content that aligns with 2D representations while maintaining compositional structure. To ensure completeness and spatial alignment of each decomposed instance, we develop a video-diffusion-based amodal completion technique that effectively handles occlusions and shadows between objects, and introduce shape prior injection to ensure spatial coherence within the scene. Experimental results demonstrate that our method produces more natural object arrangements and complete object instances suitable for interactive applications, while maintaining physical plausibility and alignment with user inputs.

Problem

• Bridging 2D and 3D generation for hierarchical scenes
• Enhancing scene editing flexibility and object diversity
• Ensuring spatial coherence and occlusion handling in 3D scenes

Innovation

• Hierarchical framework for 3D scene generation
• Video-diffusion-based amodal completion technique
• Shape prior injection for spatial coherence