GuideGround: VLM-guided Semantic Understanding and Viewpoint-aware Reasoning for 3D Visual Grounding

2026-08-01

AI Summary

This work addresses the limitations of weak semantic generalization and insufficient viewpoint-aware spatial reasoning in 3D visual grounding. To overcome these challenges, it introduces vision-language models (VLMs) to enhance open-vocabulary semantic understanding and proposes a viewpoint-specific hypothesis generation and explicit verification mechanism, thereby avoiding information loss caused by direct fusion of multi-view features. By integrating VLMs, open-ended semantic descriptions, and multi-view 3D scene modeling, the proposed method significantly outperforms existing state-of-the-art approaches on the ReferIt3D benchmark. Ablation studies further confirm the effectiveness of both semantic enhancement and viewpoint-aware reasoning components.

Abstract

3D visual grounding aims to localize the target object in a 3D scene from a natural language query, requiring both fine-grained semantic understanding and viewpoint-dependent spatial reasoning. Existing methods typically formulate semantic understanding as an auxiliary closed-set object classification task and rely on multi-view feature aggregation for viewpoint reasoning, limiting semantic generalization and weakening viewpoint-specific evidence. We observe that vision-language models naturally provide complementary capabilities through open-vocabulary semantic understanding and global scene perception. Based on this insight, we propose GuideGround, a VLM-guided framework that complements rather than replaces task-specific grounding models by leveraging VLMs for semantic enhancement and viewpoint-specific hypothesis verification. Specifically, we replace auxiliary closed-set object classification with VLM-generated object semantic descriptions to enhance semantic understanding. Meanwhile, instead of directly aggregating multi-view representations, we preserve viewpoint-specific grounding hypotheses through per-view grounding and explicitly verify them using VLMs across candidate viewpoints. Extensive experiments on the ReferIt3D benchmark demonstrate that GuideGround consistently outperforms previous state-of-the-art methods. Comprehensive ablation studies further confirm the effectiveness of both the proposed semantic understanding and viewpoint reasoning strategies.

Problem

• 3D visual grounding
• semantic understanding
• viewpoint-aware reasoning
• vision-language models
• open-vocabulary

Innovation

• vision-language models
• open-vocabulary semantic understanding
• viewpoint-aware reasoning
• 3D visual grounding
• hypothesis verification