Unposed Sparse Views Room Layout Reconstruction in the Age of Pretrain Model

2025-02-24

AI Summary

This work addresses indoor room layout estimation from uncalibrated, sparse multi-view images—bypassing conventional multi-stage geometric pipelines (e.g., camera calibration, feature matching, triangulation). We pioneer the adaptation of the 3D foundation model DUSt3R to structured planar modeling, proposing a plane-aware fine-tuning framework that directly outputs compact, globally consistent 3D layouts from uncalibrated, non-panoramic, single-view inputs via a single post-processing step. Our method integrates structured plane supervision, domain adaptation to the Structure3D dataset, and sparse multi-view geometric representation learning. Evaluated on synthetic benchmarks, it surpasses state-of-the-art methods; on real-world images—including stylistically diverse and cartoon-like scenes—it demonstrates strong robustness, significantly mitigating error accumulation. The resulting layouts exhibit uniform geometry and minimal parametric complexity.

Technology Category

• Computer Vision: 3D Computer Vision
• Planning, Routing, and Scheduling: Planning with Language Models
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation

Abstract

Room layout estimation from multiple-perspective images is poorly investigated due to the complexities that emerge from multi-view geometry, which requires muti-step solutions such as camera intrinsic and extrinsic estimation, image matching, and triangulation. However, in 3D reconstruction, the advancement of recent 3D foundation models such as DUSt3R has shifted the paradigm from the traditional multi-step structure-from-motion process to an end-to-end single-step approach. To this end, we introduce Plane-DUSt3R}, a novel method for multi-view room layout estimation leveraging the 3D foundation model DUSt3R. Plane-DUSt3R incorporates the DUSt3R framework and fine-tunes on a room layout dataset (Structure3D) with a modified objective to estimate structural planes. By generating uniform and parsimonious results, Plane-DUSt3R enables room layout estimation with only a single post-processing step and 2D detection results. Unlike previous methods that rely on single-perspective or panorama image, Plane-DUSt3R extends the setting to handle multiple-perspective images. Moreover, it offers a streamlined, end-to-end solution that simplifies the process and reduces error accumulation. Experimental results demonstrate that Plane-DUSt3R not only outperforms state-of-the-art methods on the synthetic dataset but also proves robust and effective on in the wild data with different image styles such as cartoon.

Problem

• Multi-view room layout estimation
• End-to-end single-step approach
• Handling multiple-perspective images

Innovation

• Leverages DUSt3R foundation model
• Streamlines multi-view room layout estimation
• Fine-tunes with modified objective