CIBR: Cross-modal Information Bottleneck Regularization for Robust CLIP Generalization

2025-03-31

AI Summary

This work addresses the lack of theoretical understanding of CLIP’s cross-modal generalization capability by proposing the Cross-modal Information Bottleneck (CIB) framework—the first formal information-theoretic characterization of CLIP’s implicit optimization mechanism from an information bottleneck perspective. Building on this theory, we design CIBR, the first explicit, trainable regularization method that simultaneously suppresses modality-specific redundancy and enhances semantic alignment, thereby unifying theoretical rigor with practical trainability. Evaluated on seven zero-shot image classification benchmarks and two cross-modal retrieval tasks (MSCOCO and Flickr30K), CIBR consistently outperforms standard CLIP, empirically validating the efficacy of theory-driven design. Our core contributions are threefold: (i) establishing an information-theoretic explanation for CLIP’s generalization behavior; (ii) introducing the first explicit cross-modal information bottleneck regularization paradigm; and (iii) demonstrating its consistent improvement in cross-modal generalization performance.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Information Theory
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods

Abstract

Contrastive Language-Image Pretraining (CLIP) has achieved remarkable success in cross-modal tasks such as zero-shot image classification and text-image retrieval by effectively aligning visual and textual representations. However, the theoretical foundations underlying CLIP's strong generalization remain unclear. In this work, we address this gap by proposing the Cross-modal Information Bottleneck (CIB) framework. CIB offers a principled interpretation of CLIP's contrastive learning objective as an implicit Information Bottleneck optimization. Under this view, the model maximizes shared cross-modal information while discarding modality-specific redundancies, thereby preserving essential semantic alignment across modalities. Building on this insight, we introduce a Cross-modal Information Bottleneck Regularization (CIBR) method that explicitly enforces these IB principles during training. CIBR introduces a penalty term to discourage modality-specific redundancy, thereby enhancing semantic alignment between image and text features. We validate CIBR on extensive vision-language benchmarks, including zero-shot classification across seven diverse image datasets and text-image retrieval on MSCOCO and Flickr30K. The results show consistent performance gains over standard CLIP. These findings provide the first theoretical understanding of CLIP's generalization through the IB lens. They also demonstrate practical improvements, offering guidance for future cross-modal representation learning.

Problem

• Understanding CLIP's generalization via Information Bottleneck theory
• Reducing modality-specific redundancy in cross-modal learning
• Enhancing semantic alignment between image and text features

Innovation

• Cross-modal Information Bottleneck framework
• Regularization to reduce modality-specific redundancy
• Enhanced semantic alignment across modalities