Steer Where It Matters: Token-Level Visual-Sensitivity Steering for LVLMs Hallucination Mitigation

2026-06-02

AI Summary

This work addresses the challenge of hallucination in large vision-language models (LVLMs) during text generation, where existing activation-based intervention methods suffer from signal dilution and fixed intervention strength. The authors propose a lightweight, plug-and-play token-level visual sensitivity intervention mechanism that applies fine-grained, adaptively scaled interventions only at critical positions during autoregressive decoding. This approach effectively suppresses hallucinations while preserving factual content by dynamically modulating interventions based on per-token visual sensitivity—avoiding the signal dilution caused by global averaging. Through token-level intervention vector optimization and minimal calibration training, the method is compatible with diverse LVLM architectures. Extensive experiments demonstrate significant performance gains over state-of-the-art baselines across multiple benchmarks, including POPE, AMBER, CHAIR, MMHal, and HallusionBench.

Abstract

Large vision language models (LVLMs) have made rapid advancements and are deployed across various applications, yet hallucinations remain a major challenge. Activation steering is appealing due to its minimal training overhead and controllability at inference time. However, we found that during autoregressive decoding, visual conditioning affects token prediction sparsely and locally across decoding steps, and many existing methods that average image-versus-no-image differences over the entire sequence dilute these critical signals, yielding low signal-to-noise ratio steering directions. Additionally, many existing methods apply a fixed steering strength, which misallocates the intervention budget, over-perturbs non-critical tokens, and can cause instability. To address these limitations, we propose Token-Level Visual-Sensitivity Steering (TLVS) for hallucination mitigation. Our approach first extracts token-level steering vectors and refines them, and then applies fine-grained, visual-sensitivity-adaptive steering only where it matters. This lightweight, plug-and-play mechanism requires only minimal training for calibration and can be applied across diverse vision-language models. It modulates the steering strength at each decoding step, selectively suppressing hallucination-prone spans while preserving evidence-grounded content. We evaluate TLVS on several benchmarks, including POPE, AMBER, CHAIR (COCO), MMHal, and HallusionBench, demonstrating consistent improvements over previous steering methods.

Problem

• hallucination
• activation steering
• visual-language models
• token-level intervention
• signal-to-noise ratio

Innovation

• Token-Level Steering
• Visual-Sensitivity
• Hallucination Mitigation
• Activation Steering
• Autoregressive Decoding