Learning from Fine-Grained Visual Discrepancies: Mitigating Multimodal Hallucinations via In-Context Visual Contrastive Optimization

2026-05-29

AI Summary

This work addresses multimodal hallucination in vision-language models, which often arises from insufficient explicit visual supervision and coarse-grained negative samples. To mitigate this issue, the authors propose an In-Context Visual Contrastive Optimization (IC-VCO) framework that leverages fine-grained contrastive learning across shared multi-image contexts to enable visually grounded preference optimization. Additionally, they introduce a reliability-gated Visual Contrastive Distillation (VCDist) mechanism to align model behavior during training and inference. A hard negative sample generation strategy based on semantically precise perturbations is also devised to enhance the model’s discriminative capability. Evaluated on five benchmarks, the proposed approach achieves state-of-the-art overall performance and significantly alleviates hallucination, demonstrating the effectiveness of both the IC-VCO framework and the tailored negative sample editing strategy.

Abstract

Multimodal hallucination remains a persistent challenge for Vision-Language Models (VLMs). Standard textual Direct Preference Optimization (DPO) often fails to mitigate it due to a lack of explicit visual supervision. While existing works introduce visual preference DPO by contrasting original images against negative ones, they suffer from a theoretically inconsistent objective caused by partition function mismatches and rely on coarse-grained negatives that could enable shortcut learning. In this work, we propose In-Context Visual Contrastive Optimization (IC-VCO). By placing contrastive images within a shared multi-image context, IC-VCO ensures a mathematically rigorous objective. We further introduce Visual Contrast Distillation (VCDist), an auxiliary reliability-gated regularizer that encourages consistency between multi-image contrastive training and single-image inference. Finally, we propose a contrastive sample editing strategy that generates hard negatives via precise semantic perturbations. Experiments on five benchmarks demonstrate IC-VCO's best overall performance and the effectiveness of our sample editing strategy. Code and data are available at https://github.com/OPPO-Mente-Lab/IC-VCO.

Problem

• multimodal hallucination
• Vision-Language Models
• visual preference
• contrastive learning
• fine-grained visual discrepancies

Innovation

• In-Context Visual Contrastive Optimization
• Visual Contrast Distillation
• Multimodal Hallucination
• Hard Negative Generation
• Vision-Language Models