CAER: Conflict-Aware Evidence Routing with Dual Prefix Experts for Multimodal Large Language Models

2026-07-30

AI Summary

This work addresses the susceptibility of multimodal large language models (MLLMs) to hallucination when confronted with conflicting textual and visual inputs. To mitigate this issue, the authors propose CAER, a conflict-aware generation framework that operates without fine-tuning the backbone model. CAER introduces soft text queries to retrieve evidence from frozen visual tokens, enabling fine-grained localization of conflicts, and incorporates a dual-prefix expert routing mechanism that explicitly disentangles and selects between supporting and contradictory information to guide response generation. Experimental results on the MMMC benchmark and a newly curated AgriConflict dataset demonstrate that CAER substantially enhances the consistency and reliability of open-source MLLMs in handling multimodal conflicts.

Abstract

Multimodal Large Language Models (MLLMs) have demonstrated remarkable capabilities in multimodal understanding and generation. However, when textual inputs conflict with visual evidence, they still suffer from hallucinations and produce responses inconsistent with visual content. Existing approaches mainly rely on decoding strategies, additional training, verification methods, or prompting techniques, but often lack fine-grained conflict localization and conflict-aware generation. In this work, we propose CAER, a backbone-agnostic framework for visual-language conflict detection and conflict-aware generation. CAER introduces a span-grounded evidence router that transforms claim representations into soft textual queries and retrieves corresponding evidence from frozen visual tokens, enabling fine-grained conflict estimation. Furthermore, we design a dual-prefix expert routing mechanism that learns separate experts for visually supported and contradicted inputs, enabling conflict-aware generation through explicit expert selection. Experiments on the public MMMC benchmark and our newly curated AgriConflict dataset demonstrate that CAER effectively detects visual-language conflicts and improves the reliability of open-source MLLMs without updating their backbone parameters.

Problem

• multimodal large language models
• visual-language conflict
• hallucination
• conflict detection
• evidence routing

Innovation

• conflict-aware generation
• evidence routing
• dual-prefix experts
• multimodal hallucination
• visual-language conflict