ViGiL3D: A Linguistically Diverse Dataset for 3D Visual Grounding

2025-01-02

AI Summary

Current 3D visual grounding (3DVG) models exhibit significant performance degradation under complex, out-of-distribution natural language descriptions, hindering real-world deployment. To address this, we introduce ViGiL3D—the first multilingual diagnostic dataset for 3DVG, explicitly designed to capture referential diversity and linguistic challenge. We further propose the first linguistics-informed evaluation framework for prompt diversity in 3DVG, systematically covering complex syntactic constructions, semantic phenomena, and out-of-distribution patterns in English. Leveraging linguistically motivated heuristics, structured sampling, and rigorous human validation, we generate high-quality, open-vocabulary annotations. Empirical analysis reveals severe robustness deficiencies in state-of-the-art open-vocabulary 3DVG models under linguistically complex prompts. ViGiL3D thus enables fine-grained diagnostic analysis, establishes a reproducible benchmark, and provides critical support for targeted model improvement.

Technology Category

• Natural Language Processing: Language Grounding & Multi-modal NLP
• Computer Vision: 3D Computer Vision
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

3D visual grounding (3DVG) involves localizing entities in a 3D scene referred to by natural language text. Such models are useful for embodied AI and scene retrieval applications, which involve searching for objects or patterns using natural language descriptions. While recent works have focused on LLM-based scaling of 3DVG datasets, these datasets do not capture the full range of potential prompts which could be specified in the English language. To ensure that we are scaling up and testing against a useful and representative set of prompts, we propose a framework for linguistically analyzing 3DVG prompts and introduce Visual Grounding with Diverse Language in 3D (ViGiL3D), a diagnostic dataset for evaluating visual grounding methods against a diverse set of language patterns. We evaluate existing open-vocabulary 3DVG methods to demonstrate that these methods are not yet proficient in understanding and identifying the targets of more challenging, out-of-distribution prompts, toward real-world applications.

Problem

• 3D visual localization
• natural language description
• real-world application

Innovation

• ViGiL3D Dataset
• 3D Visual Localization
• Multilingual Descriptions