Towards Visual Grounding: A Survey

2024-12-28

AI Summary

This survey addresses the lack of systematic reviews and standardized definitions in visual grounding. It comprehensively surveys the past decade’s research, unifying foundational task definitions and diverse settings—including grounding pretraining, MLLM-based grounding, and generalized/ultra-high-resolution grounding—and establishes a standardized evaluation framework. Methodologically, it synthesizes advances in deep learning, cross-modal alignment, contrastive learning, Transformers, and multimodal large language models (MLLMs), analyzing evolutionary trends and core challenges. Key contributions include: (1) constructing the most comprehensive knowledge system for visual grounding to date; (2) open-sourcing a high-quality resource repository (Awesome-Visual-Grounding); (3) proposing emerging directions such as grounded MLLMs and giga-pixel-level localization; and (4) releasing a structured knowledge graph tailored for both beginners and experts—thereby significantly advancing field standardization and sustainable development.

Technology Category

• Natural Language Processing: Language Grounding & Multi-modal NLP
• Computer Vision: Large Vision Models
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Vertical and domain-specific search

Abstract

Visual Grounding is also known as Referring Expression Comprehension and Phrase Grounding. It involves localizing a natural number of specific regions within an image based on a given textual description. The objective of this task is to emulate the prevalent referential relationships in social conversations, equipping machines with human-like multimodal comprehension capabilities. Consequently, it has extensive applications in various domains. However, since 2021, visual grounding has witnessed significant advancements, with emerging new concepts such as grounded pre-training, grounding multimodal LLMs, generalized visual grounding, and giga-pixel grounding, which have brought numerous new challenges. In this survey, we initially examine the developmental history of visual grounding and provide an overview of essential background knowledge. We systematically track and summarize the advancements and meticulously organize the various settings in visual grounding, thereby establishing precise definitions of these settings to standardize future research and ensure a fair comparison. Additionally, we delve into several advanced topics and highlight numerous applications of visual grounding. Finally, we outline the challenges confronting visual grounding and propose valuable directions for future research, which may serve as inspiration for subsequent researchers. By extracting common technical details, this survey encompasses the representative works in each subtopic over the past decade. To the best, this paper presents the most comprehensive overview currently available in the field of grounding. This survey is designed to be suitable for both beginners and experienced researchers, serving as an invaluable resource for understanding key concepts and tracking the latest research developments. We keep tracing related works at https://github.com/linhuixiao/Awesome-Visual-Grounding.

Problem

• Visual Localization
• Comprehensive Review
• Standardization

Innovation

• Comprehensive Review
• Visual Localization
• Future Research Directions