Seeing What Should Be Heard: Diagnosing and Repairing Cross-Modal Shortcuts in Omni-Modal LLMs

2026-09-29

AI Summary

This study addresses the cross-modal shortcut problem in omni-modal large language models, where the model disproportionately relies on visual inputs rather than the designated modality when answering audio questions. To this end, we propose a factorized modality diagnostic method to isolate the causal contribution of each modality, alongside a DMC-Repair training strategy that integrates cross-sample substitution, supervised fine-tuning, and reinforcement learning to provide targeted supervision and effectively suppress spurious modality correlations. Experimental results demonstrate that our approach reduces the image-induced answer effect by 59.9%, significantly mitigating shortcuts without compromising overall model performance. Furthermore, the proposed method exhibits strong generalization capabilities across diverse model architectures and novel datasets.

Abstract

Omni-modal large language models (LLMs) are expected to answer a question using the modality it explicitly refers to. However, existing training paradigms rarely verify whether models actually follow this modality, because multimodal inputs from the same sample often provide redundant evidence for the same answer. In this work, we uncover a pervasive cross-modal shortcut in omni-modal LLMs: when asked an audio-related question, models rely on the image as much as on the audio, and sometimes even more. To systematically diagnose this behavior, we introduce the Factorized Modality Diagnostic, which independently swaps audio and images between samples to isolate each modality's causal contribution. Across two model families in different settings, we find that this shortcut persists throughout supervised fine-tuning and reinforcement learning post-training, while judge-based RL may further amplify such reliance on irrelevant visual information. Based on this finding, we propose DMC-Repair, which trains models on the same kind of cross-modal swapped samples while assigning supervision according to the modality specified by the question. This prevents models from exploiting the spurious correspondence between modalities within the same clip. Experiments demonstrate that DMC-Repair reduces the image-induced share of the answer effect by 59.9%, effectively suppressing the cross-modal shortcut without compromising audio-question answering performance. The reduction in shortcut reliance generalizes across two model families and zero-shot to an unseen dataset and an unseen benchmark, and persists through subsequent post-training. Code is available at https://anonymous.4open.science/r/DMC-Repair.

Problem

• omni-modal LLMs
• cross-modal shortcuts
• modality reliance
• multimodal redundancy

Innovation

• Omni-modal LLMs
• Cross-modal shortcuts
• Factorized Modality Diagnostic
• DMC-Repair
• Causal contribution