Debias in Text, Believe Your Eyes: Text-Anchored Cross-Modal Transfer for Visual Counter-Commonsense Reasoning

2026-08-07

AI Summary

This work addresses the tendency of multimodal large language models to fail on visual counterfactual reasoning tasks not due to insufficient visual perception, but because of overreliance on linguistic priors. To mitigate this, the authors propose TACT, a two-stage post-training framework that operates without visual data: it first constructs a text-anchored dataset and then employs Fact-Frequency Distillation to quantify the strength of commonsense priors, guiding a shared language decoder to prioritize visual evidence over linguistic priors when conflicts arise. TACT establishes a novel paradigm of purely text-driven debiasing for decoders, enabling cross-modal transfer of debiasing from text to vision. The method achieves significant performance gains on multiple visual counterfactual reasoning benchmarks while preserving the model’s general capabilities.

Abstract

The visual reasoning ability of multimodal large language models (MLLMs) is crucial for downstream applications, particularly counter-commonsense reasoning, which requires models to reason beyond common assumptions. Recent studies mainly improve visual counter-commonsense reasoning by enhancing visual inputs, following the assumption that failures originate from insufficient visual grounding. However, our empirical analysis reveals that the bottleneck is not visual perception. MLLMs already capture the relevant visual evidence, and the correct answer exists in their decoding space. Instead, the shared language decoder resolves prior--evidence conflicts by favoring dominant language priors, especially for low-frequency factual scenarios. Motivated by this, we first propose a text-anchored data construction pipeline, whose core component, Fact-Frequency Distillation (FFD), estimates the prior strength of commonsense facts and distills verified counter-commonsense scenarios into a high-quality text corpus. Building upon this corpus, we introduce TACT, a text-anchored post-training framework that debiases the shared language decoder without requiring any visual training data. TACT routes evidence-following and prior-driven reasoning trajectories into different optimization stages, enabling the decoder to resolve prior--evidence conflicts. Across counter-commonsense visual benchmarks, TACT substantially improves visual reasoning while preserving general capabilities, demonstrating effective text-to-vision cross-modal transfer.

Problem

• visual counter-commonsense reasoning
• multimodal large language models
• language priors
• prior-evidence conflict
• cross-modal transfer

Innovation

• text-anchored transfer
• counter-commonsense reasoning
• language prior debiasing
• Fact-Frequency Distillation
• cross-modal reasoning