YARD: Y-Architecture Register Decoding for Efficient Hallucination Mitigation in Large Vision-Language Models

2026-05-29

AI Summary

This work addresses a critical limitation in existing training-free contrastive decoding methods for mitigating hallucinations in large vision-language models: their degradation branches either discard excessive visual information or rely on high-latency, coarse-grained image perturbations. To overcome this, the authors propose YARD, a novel framework that introduces a Y-shaped architecture with register tokens within intermediate decoder layers to construct an internal degradation branch. This design simulates localized absence of visual evidence while preserving global semantics and cross-modal alignment, all while sharing early-layer computations. Crucially, YARD avoids additional full forward passes, substantially reducing inference latency. The method achieves state-of-the-art hallucination mitigation performance across multiple generative and discriminative hallucination benchmarks.

Abstract

Contrastive decoding (CD) seeks to mitigate hallucinations in Large Vision-Language Models (LVLMs) by contrasting the output distributions of a standard model and a visually degraded model. However, existing training-free CD methods suffer from sub-optimal degraded branches: completely dropping visual tokens is too extreme and induces language hallucinations, while corrupting input images offers coarse control over visual evidence and suffers from high inference latency due to requiring two full forward passes. To address these dilemmas, we propose YARD, a training-free Y-Architecture Register Decoding framework. Motivated by the observation that reliable text-to-vision grounding predominantly emerges in the middle decoder layers, YARD constructs the degraded branch internally by sharing shallow-layer computations and branching exactly at this critical stage. For the degraded branch, YARD replaces patch-level visual tokens with register tokens, which preserve global image semantics but lack fine-grained local evidence. This image-aware yet locally under-grounded design provides a faithful contrastive signal without extreme modality mismatch, while the Y-architecture strictly avoids a costly second forward pass. Extensive experiments on generative and discriminative hallucination benchmarks demonstrate that YARD consistently achieves state-of-the-art hallucination mitigation across multiple LVLMs, alongside a significant reduction in inference latency.

Problem

• hallucination mitigation
• Large Vision-Language Models
• contrastive decoding
• visual tokens
• inference latency

Innovation

• Y-Architecture
• Register Tokens
• Contrastive Decoding
• Hallucination Mitigation
• Vision-Language Models