AuraFusion360: Augmented Unseen Region Alignment for Reference-based 360{deg} Unbounded Scene Inpainting

2025-02-07

AI Summary

To address viewpoint inconsistency and geometric distortion in 360° unbounded scene reconstruction, this paper proposes the first high-fidelity inpainting framework tailored for Gaussian splatting representations. Methodologically: (1) we introduce a novel depth-aware unknown-region masking mechanism; (2) incorporate a zero-shot adaptive guided depth diffusion prior; and (3) jointly leverage SDEdit-driven detail enhancement and spherical coordinate alignment, augmented with multi-view consistency constraints. Our contributions include: (i) establishing 360-USID—the first benchmark dataset for 360° unbounded scene inpainting; (ii) enabling end-to-end optimization for object removal and hole filling; and (iii) significantly improving perceptual quality and geometric accuracy under large viewpoint variations. All code, datasets, and video demonstrations are publicly released.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Planning, Routing, and Scheduling: Replanning and Plan Repair
• Search and Optimization: Learning to Search

Application Category

• Responsible Web: Data and user privacy-enhancing technologies for the Web
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs

Abstract

Three-dimensional scene inpainting is crucial for applications from virtual reality to architectural visualization, yet existing methods struggle with view consistency and geometric accuracy in 360{deg} unbounded scenes. We present AuraFusion360, a novel reference-based method that enables high-quality object removal and hole filling in 3D scenes represented by Gaussian Splatting. Our approach introduces (1) depth-aware unseen mask generation for accurate occlusion identification, (2) Adaptive Guided Depth Diffusion, a zero-shot method for accurate initial point placement without requiring additional training, and (3) SDEdit-based detail enhancement for multi-view coherence. We also introduce 360-USID, the first comprehensive dataset for 360{deg} unbounded scene inpainting with ground truth. Extensive experiments demonstrate that AuraFusion360 significantly outperforms existing methods, achieving superior perceptual quality while maintaining geometric accuracy across dramatic viewpoint changes. See our project page for video results and the dataset at https://kkennethwu.github.io/aurafusion360/.

Problem

• Improves 360° unbounded scene inpainting quality
• Ensures view consistency and geometric accuracy
• Introduces a novel dataset for scene inpainting

Innovation

• Depth-aware unseen mask generation
• Adaptive Guided Depth Diffusion
• SDEdit-based detail enhancement