Beyond Modality Collapse: Representations Blending for Multimodal Dataset Distillation

2025-05-16

AI Summary

Multimodal dataset distillation (MDD) suffers from modality collapse—characterized by excessive intra-modal representation concentration and inter-modal distribution misalignment—exacerbated by asymmetric cross-modal supervision in existing methods, leading to optimization bias. This work first identifies the root cause as an inherent conflict between distillation compression objectives and contrastive learning goals. To address this, we propose RepBlend: a novel framework that weakens overly strong cross-modal constraints via representation blending to enhance intra-modal diversity; introduces modality-specific projection heads and a symmetric projection trajectory matching mechanism to jointly optimize intra-modal diversity and inter-modal alignment. Evaluated on Flickr-30K and MS-COCO under the 100-sample setting, RepBlend achieves +9.4 and +6.3 absolute gains in IR@10 and TR@10, respectively, and accelerates distillation by up to 6.7×, significantly outperforming state-of-the-art methods.

Technology Category

Machine Learning: Multimodal LearningComputer Vision: Multi-modal VisionIntelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAGWeb Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web dataUser Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Multimodal Dataset Distillation (MDD) seeks to condense large-scale image-text datasets into compact surrogates while retaining their effectiveness for cross-modal learning. Despite recent progress, existing MDD approaches often suffer from extit{ extbf{Modality Collapse}}, characterized by over-concentrated intra-modal representations and enlarged distributional gap across modalities. In this paper, at the first time, we identify this issue as stemming from a fundamental conflict between the over-compression behavior inherent in dataset distillation and the cross-modal supervision imposed by contrastive objectives. To alleviate modality collapse, we introduce extbf{RepBlend}, a novel MDD framework that weakens overdominant cross-modal supervision via representation blending, thereby significantly enhancing intra-modal diversity. Additionally, we observe that current MDD methods impose asymmetric supervision across modalities, resulting in biased optimization. To address this, we propose symmetric projection trajectory matching, which synchronizes the optimization dynamics using modality-specific projection heads, thereby promoting balanced supervision and enhancing cross-modal alignment. Experiments on Flickr-30K and MS-COCO show that RepBlend consistently outperforms prior state-of-the-art MDD methods, achieving significant gains in retrieval performance (e.g., +9.4 IR@10, +6.3 TR@10 under the 100-pair setting) and offering up to 6.7$ imes$ distillation speedup.

Problem

• Addressing Modality Collapse in Multimodal Dataset Distillation
• Balancing asymmetric supervision across modalities
• Enhancing intra-modal diversity and cross-modal alignment

Innovation

• RepBlend framework enhances intra-modal diversity
• Symmetric projection trajectory balances supervision
• Modality-specific projection heads align cross-modal dynamics