ASCD: Attention-Steerable Contrastive Decoding for Reducing Hallucination in MLLM

2025-06-17

AI Summary

Multimodal large language models (MLLMs) suffer from pervasive hallucination, primarily stemming from overreliance on local visual cues. To address this, we propose Contrastive Attention Decoding (CAD), a novel paradigm that directly steers self-attention distributions—bypassing conventional logit-level post-hoc correction—and suppresses erroneous responses at the attention mechanism’s core. We first theoretically characterize VCD/ICD-style methods as implicit attention modulation, then introduce the first explicit attention-steering contrastive decoding framework. CAD comprises three key components: attention gating intervention, contrastive attention masking, and multi-granularity visual/instruction perturbation, enabling seamless integration across diverse decoder architectures. Extensive evaluation on POPE, CHAIR, and MMHal-Bench demonstrates substantial hallucination reduction while simultaneously improving standard VQA accuracy. CAD is architecture-agnostic and validated on multiple MLLMs, including Qwen-VL, LLaVA, and InternVL.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Large Vision Models
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Multimodal Large Language Model (MLLM) often suffer from hallucinations. They over-rely on partial cues and generate incorrect responses. Recently, methods like Visual Contrastive Decoding (VCD) and Instruction Contrastive Decoding (ICD) have been proposed to mitigate hallucinations by contrasting predictions from perturbed or negatively prefixed inputs against original outputs. In this work, we uncover that methods like VCD and ICD fundamentally influence internal attention dynamics of the model. This observation suggests that their effectiveness may not stem merely from surface-level modifications to logits but from deeper shifts in attention distribution. Inspired by this insight, we propose an attention-steerable contrastive decoding framework that directly intervenes in attention mechanisms of the model to offer a more principled approach to mitigating hallucinations. Our experiments across multiple MLLM architectures and diverse decoding methods demonstrate that our approach significantly reduces hallucinations and improves the performance on benchmarks such as POPE, CHAIR, and MMHal-Bench, while simultaneously enhancing performance on standard VQA benchmarks.

Problem

• Reducing hallucinations in Multimodal Large Language Models
• Addressing over-reliance on partial cues in MLLM responses
• Improving attention dynamics to mitigate incorrect outputs

Innovation

• Directly intervenes in attention mechanisms
• Steers internal attention dynamics
• Contrastive decoding reduces hallucinations