CARGO-VL: Counterfactual Arbitration with Risk-Constrained Group Optimization for Vision-Language Models

2026-08-05

AI Summary

This work addresses the challenge that existing vision-language models struggle to reliably assess information credibility and appropriately abstain when image and text inputs are conflicting or simultaneously incorrect. To tackle this, the authors propose the CARGO-VL framework, which jointly models four evidence states—aligned, image-correct, text-correct, and both-incorrect—through a group-wise relative optimization objective and a primal-dual controller. This design enables coordinated enforcement of counterfactual consistency, modality-equivariance, and risk-constrained abstention. The approach integrates counterfactual data augmentation with a multi-condition transition reward mechanism and introduces the XMC conflict training set. Evaluated on CMC-Bench and Modality-Bias benchmarks, CARGO-VL significantly enhances conflict-handling capability, reduces unfounded responses, and improves balance in modality usage, outperforming pointwise optimization baselines.

Abstract

Vision-language systems combine images with retrieved text, but these sources can disagree or jointly fail to support an answer. Reliable models must identify the trustworthy source and abstain when neither is adequate. Existing post-training objectives score instances independently and therefore do not enforce coherent behavior under counterfactual evidence changes. We introduce CARGO-VL, a group-relative framework that optimizes matched variants covering aligned, image-correct, text-correct, and both-wrong (A/V/T/N) evidence states as one bundle. Its objective couples condition-wise correctness with transition rewards for answer invariance, source equivariance, and answer-to-abstention switching, while a primal-dual controller balances unsafe answers against excessive deferral. We also contribute XMC (eXtended Modal Conflict), a four-condition conflict training resource, and evaluate transfer on CMC-Bench and Modality-Bias. Across multiple seeds, CARGO-VL improves conflict handling, unsupported-answer avoidance, and modality balance over pointwise baselines. Ablations identify complementary benefits from relational transition signals and adaptive risk control, supporting counterfactual consistency as a practical objective for reliable multimodal evidence arbitration.

Problem

• vision-language models
• modal conflict
• counterfactual consistency
• evidence arbitration
• abstention

Innovation

• counterfactual consistency
• group-relative optimization
• risk-constrained arbitration
• multimodal conflict handling
• adaptive deferral control