RelationVGGT: Visual Geometry Transformers for 3D Spatial Relation Segmentation

2026-09-30

AI Summary

This study addresses the reliance on camera poses and per-scene optimization in multi-view 3D spatial relationship segmentation by proposing RelationVGGT, the first pose-free feedforward framework for this task. The method integrates features from vision and 3D geometry foundation models, employing a relation Transformer to perform cross-view subject-conditioned reasoning, thereby enabling accurate target object segmentation without requiring known camera poses. Furthermore, an automated data annotation pipeline built upon ScanNet++ and large-scale models is constructed to facilitate scalable training. Experimental results demonstrate that, under unseen categories, the proposed framework accurately performs open-vocabulary 3D spatial relationship segmentation using only visual subject specifications and text queries.

Abstract

Recent advances in 3D reconstruction have progressed from per-scene optimization to feed-forward inference, and semantic scene understanding has followed suit -- yet existing methods remain confined to object-centric perception, neglecting spatial relations between objects. We formulate 3D spatial relation segmentation in a feed-forward, pose-free multi-view setting: given a visually specified subject and a relational text query, the model segments the target across views without receiving its category name. To this end, we propose RelationVGGT, a novel feed-forward framework that integrates semantic features from a visual foundation model with geometry-aware representations from a 3D geometry foundation model and leverages a relation transformer for subject-conditioned, cross-view relation prediction -- requiring neither per-scene optimization nor known camera poses. We additionally provide a fully automated annotation pipeline built on ScanNet++ with VLMs and LLMs, enabling scalable training data generation for this new task.

Problem

• 3D spatial relation segmentation
• multi-view
• pose-free
• feed-forward inference
• semantic scene understanding

Innovation

• 3D Spatial Relation Segmentation
• Feed-forward Framework
• Pose-free Multi-view
• Visual Geometry Foundation Models
• Automated Annotation Pipeline