DICA: Dual-Indicator Guided Contrastive Alignment in Multimodal Large Language Models

2026-07-26

AI Summary

This work addresses the prevalent issue of hallucination in multimodal large language models during reasoning, which often stems from attention drift and insufficient utilization of visual evidence. To mitigate this, the authors propose a dual-metric–driven dynamic intervention mechanism that continuously monitors model behavior using Visual Attention Entropy (VAE) and Output-Image Correlation (OIC). Upon detecting anomalies indicative of multimodal misalignment, the system triggers a contrastive alignment strategy to restore visual grounding. This approach uniquely integrates information-theoretic metrics with dynamic contrastive learning to precisely identify and correct diverse failure modes in multimodal reasoning. Evaluated across multiple benchmarks, the method significantly reduces hallucination rates and enhances the reliability of model inference without compromising fluency or generality.

Abstract

Human visual reasoning typically follows a coarse-to-fine attention process, starting from global scene understanding and gradually focusing on question-relevant regions. However, multimodal large language models may deviate from this pattern due to attention drift and the underutilization of visual evidence, which can lead to hallucinations. To mitigate these issues, this study proposes a Dual-Indicator Guided Contrastive Alignment (DICA), which tracks two information-theoretic indicators during inference: Visual Attention Entropy (VAE), which reflects the concentration of visual attention, and Output Image Correlation (OIC), which measures the dependence of generated outputs on the visual input. An abnormal increase in VAE or a decrease in OIC corresponds to different failure modes, which trigger targeted contrastive alignment to restore visual grounding. Experimental results across multiple benchmarks demonstrate that DICA consistently outperforms existing approaches and substantially reduces hallucinations, highlighting the effectiveness of indicator-driven intervention in improving multimodal inference reliability. The code is publicly available at https://github.com/BGWH123/DICA/.

Problem

• multimodal large language models
• attention drift
• visual grounding
• hallucinations
• visual reasoning

Innovation

• Dual-Indicator Guidance
• Contrastive Alignment
• Visual Attention Entropy
• Output Image Correlation
• Multimodal Hallucination Mitigation