IPFormer: Visual 3D Panoptic Scene Completion with Context-Adaptive Instance Proposals

2025-06-25

AI Summary

Current panoptic scene completion (PSC) from monocular images remains underexplored, and existing transformer-based semantic scene completion (SSC) methods rely on static, pre-learned queries, lacking contextual adaptability. To address this, we propose the first image-input context-adaptive instance proposal generation framework for PSC. Our core innovation is a dynamic instance query initialization mechanism that generates instance proposals in real time—both during training and inference—conditioned on image context. A transformer encoder-decoder jointly models 3D volumetric, semantic, and instance-level relationships. Our method achieves significant improvements over state-of-the-art: +18.65% average Panoptic Quality (PQ†) on thing classes and +14× faster inference speed. This establishes a new paradigm for efficient, accurate 3D panoptic scene understanding—particularly suitable for resource-constrained platforms such as mobile robots.

Technology Category

• Computer Vision: Scene Analysis & Understanding
• Search and Optimization: Learning to Search
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Personalized, context-aware and across-device search
• Systems and Infrastructure for Web, Mobile and WoT: Location- and context-aware Web and WoT applications and services
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Semantic Scene Completion (SSC) has emerged as a pivotal approach for jointly learning scene geometry and semantics, enabling downstream applications such as navigation in mobile robotics. The recent generalization to Panoptic Scene Completion (PSC) advances the SSC domain by integrating instance-level information, thereby enhancing object-level sensitivity in scene understanding. While PSC was introduced using LiDAR modality, methods based on camera images remain largely unexplored. Moreover, recent Transformer-based SSC approaches utilize a fixed set of learned queries to reconstruct objects within the scene volume. Although these queries are typically updated with image context during training, they remain static at test time, limiting their ability to dynamically adapt specifically to the observed scene. To overcome these limitations, we propose IPFormer, the first approach that leverages context-adaptive instance proposals at train and test time to address vision-based 3D Panoptic Scene Completion. Specifically, IPFormer adaptively initializes these queries as panoptic instance proposals derived from image context and further refines them through attention-based encoding and decoding to reason about semantic instance-voxel relationships. Experimental results show that our approach surpasses state-of-the-art methods in overall panoptic metrics PQ$^dagger$ and PQ-All, matches performance in individual metrics, and achieves a runtime reduction exceeding 14$ imes$. Furthermore, our ablation studies reveal that dynamically deriving instance proposals from image context, as opposed to random initialization, leads to a 3.62% increase in PQ-All and a remarkable average improvement of 18.65% in combined Thing-metrics. These results highlight our introduction of context-adaptive instance proposals as a pioneering effort in addressing vision-based 3D Panoptic Scene Completion.

Problem

• Addresses vision-based 3D Panoptic Scene Completion (PSC) using images.
• Overcomes static query limitations with dynamic context-adaptive instance proposals.
• Improves scene understanding by integrating instance-level semantic and geometric reasoning.

Innovation

• Context-adaptive instance proposals for scene completion
• Attention-based encoding and decoding for refinement
• Dynamic instance proposals from image context