From Suppression to Repair: Mitigating Object Hallucination in Large Vision-Language Models via Localized Distribution Alignment

2026-10-08

AI Summary

This study addresses the issue that existing object hallucination mitigation methods in large vision-language models often compromise their multimodal capabilities. To this end, we propose ResOT, a training-free framework that shifts the paradigm from suppression to rectification by correcting representations during inference via local distribution alignment. Specifically, ResOT projects hallucinatory directions onto a residual subspace and employs Gaussian optimal transport combined with adaptive token state control to achieve low-intrusiveness optimization. Experimental results demonstrate that ResOT significantly reduces object hallucinations while effectively enhancing image captioning quality and performance across diverse multimodal benchmarks.

Abstract

Object hallucination remains a major obstacle for large vision-language models (LVLMs) to generate reliable content. An intuitive mitigation strategy is to suppress hallucination-related components in hidden representations. However, these components may also contain useful information, and suppressing them can weaken the model's multimodal capabilities. In this paper, we propose ResOT, a training-free method that repairs representations at inference time through localized distribution alignment. Specifically, ResOT projects dominant hallucinated directions away from the faithful subspace, forming a low-dimensional residual subspace for intervention. Within this subspace, ResOT uses Gaussian optimal transport (OT) to align the hallucinated distribution with the faithful one. The resulting map defines repair targets with minimal changes to the original representations. At inference, ResOT adaptively controls how far each token state moves toward its OT target. Experiments on three representative LVLMs show that ResOT substantially reduces object hallucination while improving image caption quality and multimodal performance across multiple benchmarks. Code will be released.

Problem

• Object Hallucination
• Large Vision-Language Models
• Representation Repair
• Multimodal Capability

Innovation

• Object Hallucination
• Localized Distribution Alignment
• Optimal Transport
• Training-free
• Large Vision-Language Models