Chaos in the Text: Revealing the Modality Preference in Mixed-Modality Retrievers

2026-10-08

AI Summary

This study addresses the performance instability in mixed-modality retrieval, where results fluctuate drastically with varying modality compositions and irrelevant text induces a "textual chaos" effect causing severe degradation. By revealing this modality preference phenomenon, we propose Trident, a method built upon dense retrievers and CLIP/VLM architectures. Specifically, we introduce a novel multi-positive-sample InfoNCE optimization strategy that effectively balances multi-perspective representations to eliminate retrieval bias. The proposed approach significantly enhances the robustness of mixed-modality retrieval by reducing model sensitivity to textual interference, while simultaneously improving single-modality retrieval performance.

Abstract

Dense retrievers have made significant progress on text and image corpora, but whether these capabilities extend reliably to mixed corpora containing text, image, and fused text-image documents remains unclear. In this paper, we systematically examine retrievers across architectures and find that their performance is highly sensitive to modality composition. As image documents are progressively replaced with semantically corresponding text representations, retrieval performance follows a pronounced V-shaped curve, remaining strong on single-modality corpora but degrading substantially when modalities coexist. In particular, irrelevant text causes more severe degradation than an equal number of irrelevant images, a phenomenon we term Chaos in the Text. Further analysis reveals modality preference, whereby text representations receive systematically higher similarity scores, allowing irrelevant text to outrank relevant images. To mitigate this bias, we introduce Trident, which constructs text, image, and fused text-image views of each document as co-equal positives and jointly optimizes relevance discrimination and positive-view balance through Multi-Positive View InfoNCE. Experiments across visual document and natural image benchmarks show that trident improves mixed-modality retrieval on both CLIP-based and VLM-based architectures, reduces sensitivity to modality composition and text distractors, and increases average single-modality retrieval performance.

Problem

• mixed-modality retrieval
• modality preference
• dense retrievers
• Chaos in the Text

Innovation

• Mixed-Modality Retrieval
• Modality Preference
• Trident
• Multi-Positive View InfoNCE
• Chaos in the Text