🤖 AI Summary
This work addresses the degradation of cross-modal alignment in multimodal embedding spaces caused by modality bias and noisy supervision signals. To mitigate these issues, the authors propose Symmetric Nucleus Sampling (SNS) to refine training pairs and introduce an Expert Embedding Engine (EEE) that fuses representations from multiple experts. Furthermore, a bias-aware objective function and a projection network are jointly optimized to enhance embedding learning. The proposed approach substantially narrows the modality gap, achieving an average reduction of over 90%. Empirical evaluation demonstrates that the resulting embeddings significantly outperform those generated by hierarchical sampling and conventional curation baselines across downstream tasks.
📝 Abstract
Shared embedding spaces are widely used for multimodal search and data curation. In practice, two problems often limit how well this works. First, embeddings can reflect modality more than meaning, so examples cluster by input type even when the underlying content matches. Second, the paired supervision used to train these spaces is often noisy. When we blend many heterogeneous, human-labeled datasets, these issues reinforce each other and degrade cross-modal retrieval. We present a framework that improves alignment by acting on both the training pairs and the embedding model. Symmetric Nucleus Subsampling (SNS) refines training pairs by trimming raw inputs and annotations to the portions that best support each other. Expert Embedding Engine (EEE) combines complementary embedding experts using a learned projection network, together with a bias-aware objective that reduces modality-driven separation in the embedding space. We demonstrate that this approach collapses the modality gap by over 90% on average vs base embedding experts and is a strong data curator, with datablends from our method outperforming stratified sampling and traditional curation baselines in downstream model performance.