vision-language grounding

The spatial referring expression grounding (REG) field lacks a systematic survey of Transformer-based approaches, benchmark datasets, evaluation metrics, and industrial applicability. Method: This paper presents the first comprehensive review of Transformer-based spatial REG research from 2018 to 2025, covering model architectures (e.g., cross-modal attention, multimodal representation learning), mainstream benchmarks (RefCOCO series, G-Ref), and evaluation protocols (IoU, Acc@0.5). Through structured comparative analysis, it traces technical evolution—from single-stage alignment to hierarchical reasoning, and from reliance on synthetic data to generalization in real-world scenarios. Contribution/Results: The paper proposes an industrial deployment guideline with best practices, offering theoretical foundations and methodological insights for developing robust, interpretable, and production-ready vision-language alignment models. It bridges academic advances with practical engineering requirements, facilitating reproducible, scalable, and trustworthy spatial REG systems.

This paper systematically surveys visual grounding—the capability of vision-language models (VLMs) to precisely localize image regions corresponding to textual descriptions. Adopting a comprehensive literature review methodology, it analyzes prevailing techniques, application scenarios, and evaluation frameworks using standard benchmarks (e.g., RefCOCO, Flickr30K Entities) and metrics. The study establishes the first holistic research framework for visual grounding, explicitly characterizing its definition, methodologies, evaluation protocols, and open challenges—while uncovering its intrinsic connections to multimodal chain-of-thought reasoning and inferential capacity. Key limitations are identified in fine-grained localization accuracy, cross-dataset generalization, and model interpretability. To address these, the paper proposes three future directions: integrating explicit spatial modeling, reasoning-guided grounding, and unified evaluation standards. This work provides both theoretical foundations and practical guidance for advancing fine-grained cross-modal understanding.

This work addresses the challenge robots face in accurately grounding natural language instructions to target objects due to the absence of explicit spatial relations in 3D scene graphs. To this end, the authors propose an end-to-end object grounding framework that leverages a vision-language model (VLM) to generate either open- or closed-vocabulary spatial relation edges from robot-captured images and integrates them into the 3D scene graph, thereby enhancing the large language model’s (LLM) ability to localize referred objects. This study presents the first systematic evaluation of the benefits of open-vocabulary spatial relations for language-guided object grounding. Experimental results demonstrate that explicitly modeling spatial relations significantly improves LLM performance, and while VLMs can effectively generate open-vocabulary relations, their advantage over closed-vocabulary counterparts remains limited.

This work addresses audio-guided image segmentation—localizing target objects in images directly from raw speech instructions, without relying on intermediate speech-to-text conversion. Methodologically, we propose an end-to-end audio-visual alignment paradigm and introduce the first dedicated audio grounding dataset covering diverse object categories and multilingual accents. We adopt an audio-visual contrastive learning framework to jointly optimize speech embedding and visual encoders, and benchmark our approach using state-of-the-art cross-modal pre-trained models. Experiments demonstrate that our method achieves accuracy comparable to or exceeding text-mediated baselines, with significantly improved robustness under accent variation and low-resource language conditions. These results validate the feasibility and practicality of direct, text-free speech-vision alignment for grounded visual understanding.

This work addresses the challenge of simultaneously achieving structural precision, semantic interpretability, and identity controllability in existing 3D/4D scene representations. We propose “Scene Language”—a unified 3D/4D scene representation framework that integrates executable program structures, natural-language semantic tokens, and visual identity embeddings. To our knowledge, this is the first method enabling zero-shot cross-modal reasoning: without fine-tuning, it directly synthesizes structured scene programs from pretrained language models and vision encoders, while explicitly modeling hierarchical relationships to support fine-grained editing. The representation is renderer-agnostic, interfacing seamlessly with traditional, neural, and hybrid renderers to produce high-fidelity images. Experiments demonstrate significant improvements over baselines—including scene graphs—on complex scene generation tasks, achieving breakthroughs in fidelity, controllability, and editability.

To address the challenge of multi-step localization and navigation guided by task-oriented language instructions in real-world indoor 3D environments, this paper introduces the first sequential 3D vision-language grounding task. We present SG3D, a large-scale, multi-step, human-verified dataset comprising 22K tasks and 112K steps, capturing fine-grained action–target temporal relations in daily activities. We propose SG-LLM, a stepwise grounding framework that jointly leverages RGB-D scene representations and the incremental reasoning capabilities of large language models to achieve dynamic, context-aware vision-language alignment. Comprehensive evaluation on the SG3D benchmark reveals that existing methods suffer from limited multi-step contextual modeling, whereas SG-LLM achieves substantial improvements—+18.7% in sequential grounding accuracy and +22.3% in navigation success rate. This work establishes a new paradigm for task-level semantic understanding and execution in embodied agents.

This work addresses the significant degradation in grounding performance of multimodal large language models (MLLMs) in crowded scenes, where occlusion and small objects impair visual semantics. To mitigate this issue, the authors propose a novel language-guided semantic cue enhancement mechanism that leverages the inherent robustness of linguistic expressions to visual degradation. Specifically, semantic cues are extracted from the MLLM’s visual pathway and fused with language priors derived from text embeddings, thereby enriching the visual feature representations with linguistically grounded semantic information. Evaluated on challenging crowded scenarios involving occlusion and small objects, the proposed method substantially outperforms existing baselines, demonstrating the effectiveness of language-guided mechanisms in enhancing both the robustness and accuracy of visual grounding.

This work addresses the unclear mechanisms by which current vision-language models associate spatial relationships with object attributes, particularly the lack of understanding regarding how these models internally process spatial information. Through representational analysis, disentanglement of spatial relations, and enhancement of global visual tokens, the study systematically evaluates the contribution of individual components to spatial reasoning. It reveals, for the first time, that the visual encoder plays a dominant role in spatial reasoning: its output encodes global spatial signals distributed broadly across all image tokens—including background regions—rather than being confined to object-centric areas. Leveraging this insight, augmenting the visual encoder’s global spatial representations substantially improves spatial reasoning performance on natural images, challenging the conventional paradigm that focuses exclusively on object regions.

Current vision-language models exhibit limitations in complex spatial reasoning tasks, particularly in modeling 2D and 3D spatial relationships. This work proposes GR3D, a unified framework that integrates explicit and implicit 2D localization with monocular 3D localization to enable a spatial chain-of-thought process from 2D perception to 3D reasoning. GR3D innovatively treats localization as an inductive bias and incorporates region-guided generation, region token insertion, camera-intrinsic-aware normalization, dense geometric supervision, and multimodal joint training to significantly enhance spatial understanding. Experimental results demonstrate consistent performance gains across multiple spatial reasoning benchmarks—both with and without explicit localization annotations—validating that localization capability effectively facilitates general-purpose spatial reasoning.

This work addresses the challenge of precise visual grounding in multimodal large language models, which often suffer from entity hallucination, relational misalignment, or skipped reasoning steps during complex visual reasoning. To tackle this, the paper introduces a novel approach that constructs structured hard-negative reasoning paths via scene graphs, simulating four types of visual grounding failures through controlled structural interventions. By integrating Direct Preference Optimization (DPO) with explicit supervision over the reasoning process, the method departs from conventional preference alignment paradigms that rely on textual perturbations or answer-conditioned signals. This enables fine-grained, structurally faithful multimodal reasoning. Evaluated across seven visual reasoning benchmarks, the proposed model demonstrates significant improvements in both answer accuracy and reasoning faithfulness, validating the effectiveness of the grounding-aware alignment mechanism.

This work addresses the underperformance of multimodal large language models on fine-grained visual reasoning tasks, which stems primarily from their overreliance on linguistic priors during instruction tuning at the expense of visual information. The authors propose an innovative approach that reformulates classic self-supervised vision tasks—such as rotation prediction and color matching—into image-instruction-answer triplets, integrating them into the visual instruction tuning process via natural language instructions. By merely adjusting the training data distribution with just 3%–10% visually grounded instructions, the method effectively steers models to base their responses on visual evidence. This strategy requires no architectural modifications or additional training stages, yet consistently yields significant performance gains on vision-centric benchmarks across multiple models.