Retrieval Visual Contrastive Decoding to Mitigate Object Hallucinations in Large Vision-Language Models

2025-05-26

AI Summary

Object hallucination (OH) remains a pervasive issue in large vision-language models (VLMs), where models generate semantically inconsistent or nonexistent objects. To address this, we propose Retrieval-based Vision-Contrastive Decoding (RVCD), a zero-shot, inference-time decoding method that suppresses erroneous object generation without fine-tuning. RVCD introduces cross-modal contrastive decoding at the logit level by jointly incorporating positive and negative image representations, and leverages single-concept AI-generated images as explicit, controllable visual references. It synergistically combines cross-modal retrieval with logit-level intervention to refine the generation process. Experiments across multiple OH benchmarks demonstrate that RVCD significantly outperforms existing decoding-based approaches, reducing hallucination rates by an average of 32.7% while preserving text quality and diversity. This work establishes a new paradigm for reliable reasoning in VLMs through controllable, retrieval-augmented, contrastive decoding.

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Despite significant advancements in Large Vision-Language Models, Object Hallucination (OH) remains a persistent challenge. Building upon prior studies on contrastive decoding that address this issue without requiring additional model training, we introduce RVCD (Retrieval Visual Contrastive Decoding), an advanced method to suppress OH. RVCD leverages both negative and positive images at the logit level, explicitly referencing AI-generated images designed to represent a single concept. Our approach demonstrates substantial improvements over existing decoding-based methods.

Problem

• Mitigating object hallucinations in vision-language models
• Improving decoding without additional model training
• Leveraging negative and positive images for contrast

Innovation

• Uses contrastive decoding without extra training
• Leverages negative and positive images
• References AI-generated single-concept images