Disentangling Semantic Attention from Structural Bias in the Attention Manifold

2026-07-27

AI Summary

This work addresses the "visual attention trap" in multimodal large language models, where excessive focus on semantically irrelevant visual tokens leads to overreliance on linguistic priors and neglect of actual visual evidence, thereby inducing multimodal hallucinations. The study formulates this issue for the first time as a global, structural textual bias acting on visual features and introduces SPAR—a training-free, plug-and-play method that dynamically corrects text-induced attention biases during inference. SPAR achieves this through saliency-guided structural noise purification and adaptive attention reallocation. Experiments demonstrate that SPAR substantially mitigates hallucinations and enhances visual grounding across multiple benchmarks, with negligible computational overhead.

Abstract

The empirical success of attention mechanism in Multimodal Large Language Models (MLLMs) often obscures its inherent, subtle flaws. Specifically, MLLMs consistently exhibit disproportionate attention toward certain semantically uninformative visual tokens, a phenomenon termed "register" or "Visual Attention Sinks." While existing inference intervention methods attempt to identify these sink tokens and redistribute their attention weights, such approaches typically treat these tokens in isolation and suffer from computational inefficiency. Instead, we reframe this phenomenon as a generalized textual bias exerted over visual features that extends beyond isolated sink tokens. From this perspective, a pervasive structural bias leads to the dilution of the semantic visual signal, precipitating multimodal hallucinations as the model prioritizes linguistic priors over valid visual evidence. To address this limitation, we introduce Saliency-guided Purification and Adaptive Redistribution (SPAR), a training-free, plug-and-play intervention. SPAR mitigates this generalized textual bias by purifying structural noise and subsequently redistributing the reclaimed attention budget to the most informative visual regions. Comprehensive evaluations across a diverse spectrum of hallucination benchmarks demonstrate that SPAR effectively restores authentic visual grounding with negligible computational overhead.

Problem

• multimodal hallucination
• visual attention sinks
• structural bias
• semantic disentanglement
• attention mechanism

Innovation

• attention manifold
• structural bias
• visual attention sinks
• multimodal hallucination
• training-free intervention