Robust Grounding with MLLMs against Occlusion and Small Objects via Language-guided Semantic Cues

2026-04-27

AI Summary

This work addresses the significant degradation in grounding performance of multimodal large language models (MLLMs) in crowded scenes, where occlusion and small objects impair visual semantics. To mitigate this issue, the authors propose a novel language-guided semantic cue enhancement mechanism that leverages the inherent robustness of linguistic expressions to visual degradation. Specifically, semantic cues are extracted from the MLLM’s visual pathway and fused with language priors derived from text embeddings, thereby enriching the visual feature representations with linguistically grounded semantic information. Evaluated on challenging crowded scenarios involving occlusion and small objects, the proposed method substantially outperforms existing baselines, demonstrating the effectiveness of language-guided mechanisms in enhancing both the robustness and accuracy of visual grounding.

Technology Category

Natural Language Processing: Language Grounding & Multi-modal NLPMachine Learning: Large Multimodal Models (LMMs)Computer Vision: Language and Vision

Application Category

Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMsUser Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendationSearch and Retrieval-Augmented AI: Large language models for search

Abstract

While Multimodal Large Language Models (MLLMs) have enhanced grounding capabilities in general scenes, their robustness in crowded scenes remains underexplored. Crowded scenes entail visual challenges (i.e., occlusion and small objects), which impair object semantics and degrade grounding performance. In contrast, language expressions are immune to such degradation and preserve object semantics. In light of these observations, we propose a novel method that overcomes such constraints by leveraging Language-Guided Semantic Cues (LGSCs). Specifically, our approach introduces a Semantic Cue Extractor (SCE) to derive semantic cues of objects from the visual pipeline of an MLLM. We then guide these cues using corresponding text embeddings to produce LGSCs as linguistic semantic priors. Subsequently, they are reintegrated into the original visual pipeline to refine object semantics. Extensive experiments and analyses demonstrate that incorporating LGSCs into an MLLM effectively improves grounding accuracy in crowded scenes.

Problem

• grounding
• occlusion
• small objects
• crowded scenes
• robustness

Innovation

• Language-Guided Semantic Cues
• Multimodal Large Language Models
• Semantic Cue Extractor
• Robust Grounding
• Crowded Scenes