Self-Improving Small Object Grounding in LVLMs

2026-05-31

AI Summary

This work addresses the challenge of inaccurate small-object localization in large vision-language models (LVLMs) under zero-shot settings. The authors propose the Attention-based Candidate Selection (ACS) framework, which evaluates candidate bounding box quality by analyzing internal attention patterns of the LVLM. Two variants are introduced: ACS-Learned, a lightweight IoU regressor, and ACS-Free, a completely training-free strategy based on attention entropy. The study further uncovers, for the first time, the critical role of specific Transformer layers and attention heads in small-object localization, endowing the method with both self-improvement capability and interpretability. Experiments on COCO and Objects365 demonstrate up to a 19% improvement in small-object localization performance, with ACS-Free establishing a new state-of-the-art among training-free approaches.

Abstract

Can internal attention patterns in Large Vision Language Models (LVLMs) identify reliable small-object boxes without fine-tuning? In this work, we provide an affirmative answer. Attention structure in LVLMs encodes grounding quality-a lightweight IoU regressor trained solely on attention maps achieves strong IoU prediction (Pearson r > 0.67). This regressor powers the regressor-based variant of our Attention-based Candidate Selection (ACS) framework, called ACS-Learned, which selects the best box from multiple sampled candidates to improve object grounding. By analyzing what the regressor learns, we reveal which transformer layers and heads are most critical and derive ACS-Free: a training-free selector that ranks candidates by attention entropy on these discriminative heads, with no learned component at inference. Experiments on COCO and Objects365 demonstrate up to 19% self-improvement on small object localization, with ACS-Free ranking best among all training-free methods, demonstrating that useful attention structure improves both localization reliability and interpretability in LVLMs.

Problem

• small object grounding
• Large Vision Language Models
• object localization
• attention patterns
• self-improvement

Innovation

• small object grounding
• attention-based selection
• training-free method
• IoU regression
• LVLM interpretability