HAFI-VLM: A Frequency Perspective for Diagnosing and Enhancing Visual Perception in Vision-Language Models

2026-08-03

AI Summary

Existing vision-language models struggle with reliable fine-grained visual understanding due to the rigid spectral response of their pretrained visual encoders, which impedes dynamic extraction of frequency information tailored to task demands. To address this limitation, this work proposes a Hierarchical Adaptive Frequency Injection (HAFI) mechanism that leverages text-modulated cross-attention to inject complementary low-, mid-, and high-frequency visual evidence across multiple encoder layers. Additionally, a vision-enhanced adapter is introduced to optimize the large language model’s utilization of enriched visual tokens. This approach uniquely disentangles spectral allocation from semantic attention, achieving significant improvements on LLaVA-1.5 and Qwen2.5-VL in general visual question answering, text-dense comprehension, and hallucination mitigation—outperforming current representation-enhancement and high-resolution methods without requiring additional high-resolution encoding.

Abstract

Vision-language models (VLMs) remain unreliable when predictions require fine-grained visual evidence. We identify a previously overlooked cause: spectral response rigidity. Despite substantial frequency variation across images and tasks, pretrained vision encoders exhibit persistent, encoder-specific layerwise spectral profiles that change only marginally under downstream fine-tuning. Since pretrained vision encoders only receive images, they cannot adapt spectral extraction to the evidence required by the current query. We therefore propose HAFI-VLM, which introduces a task-conditioned frequency pathway while preserving the pretrained semantic representation. Hierarchical Adaptive Frequency Injection (HAFI) retrieves complementary low-, mid-, and high-frequency evidence at multiple encoder depths using text-modulated, spatially aligned cross-attention. A Visual Enrichment Layer Adapter further recalibrates shallow LLM attention to effectively utilize the enriched visual tokens. Experiments on LLaVA-1.5 and Qwen2.5-VL demonstrate consistent improvements in general VQA, text-rich understanding, and hallucination robustness, outperforming representation-level enhancement methods and most resolution- or cropping-based approaches without additional high-resolution encoding. Mechanistic analyses show that HAFI restores task-dependent spectral allocation while retaining semantic attention, establishing frequency enrichment as a distinct and effective route for improving VLM perception.

Problem

• vision-language models
• spectral response rigidity
• fine-grained visual perception
• frequency adaptation
• visual hallucination

Innovation

• frequency modulation
• vision-language models
• spectral response rigidity
• adaptive frequency injection
• cross-attention