Defurnishing with X-Ray Vision: Joint Removal of Furniture from Panoramas and Mesh

2025-06-05

AI Summary

This work addresses the joint furniture removal task for single-view panoramic images and textured meshes. We propose a cross-modal guidance method leveraging a Simplified Furniture-Free Mesh (SDM) as a geometric prior. Specifically, depth and normal maps are rendered from the SDM and processed with Canny edge detection; these geometric cues are then integrated into multi-view panoramic image inpainting via ControlNet. Concurrently, mesh topology is optimized and planar structures are extended to ensure geometric consistency. Our key contribution is the first explicit integration of the SDM as a “geometric X-ray” into the image generation pipeline, effectively mitigating structural incompleteness caused by single-view occlusions. Experiments demonstrate that our method produces high-resolution, geometrically consistent furniture-free panoramas and textured meshes, significantly outperforming NeRF-based approaches (which suffer from blurriness and low resolution) and RGB-D inpainting methods (prone to hallucination) in both detail fidelity and structural plausibility.

Technology Category

Computer Vision: Diffusion Models for VisionIntelligent Robots: Multimodal Perception & Sensor FusionMachine Learning: Large Multimodal Models (LMMs)

Application Category

Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphsSearch and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAGWeb Mining and Content Analysis: Large pretrained models with web data

Abstract

We present a pipeline for generating defurnished replicas of indoor spaces represented as textured meshes and corresponding multi-view panoramic images. To achieve this, we first segment and remove furniture from the mesh representation, extend planes, and fill holes, obtaining a simplified defurnished mesh (SDM). This SDM acts as an ``X-ray'' of the scene's underlying structure, guiding the defurnishing process. We extract Canny edges from depth and normal images rendered from the SDM. We then use these as a guide to remove the furniture from panorama images via ControlNet inpainting. This control signal ensures the availability of global geometric information that may be hidden from a particular panoramic view by the furniture being removed. The inpainted panoramas are used to texture the mesh. We show that our approach produces higher quality assets than methods that rely on neural radiance fields, which tend to produce blurry low-resolution images, or RGB-D inpainting, which is highly susceptible to hallucinations.

Problem

• Remove furniture from indoor panoramas and meshes
• Generate simplified defurnished mesh as guidance
• Improve inpainting quality using geometric control signals

Innovation

• Segment and remove furniture from mesh
• Use SDM as X-ray for scene structure
• ControlNet inpainting guided by Canny edges