Beyond Objective Expressivity: Geometry Preservation in Multimodal Contrastive Learning

2026-07-20

AI Summary

When extending multimodal contrastive learning from bimodal to trimodal or higher settings, the condition number of encoder Jacobian matrices deteriorates, often leading to representation collapse or degraded alignment performance. This work proposes a Geometry-Preserving Encoder (GPE) that jointly enhances the geometric stability of the encoder and its capacity to express the target objective through Jacobian regularization, LeakyReLU activations, and residual connections. Moving beyond prior approaches that focus solely on loss function design, GPE reveals the critical role of encoder geometry in multimodal alignment. Experiments demonstrate that GPE significantly improves retrieval and linear probing performance on both synthetic benchmarks and four real-world datasets, while also exhibiting strong robustness under missing-modality scenarios.

Abstract

Contrastive learning is increasingly moving toward settings with three or more modalities instead of image-text pairs. Yet, extending models from pairwise to higher-order multimodal alignment can introduce optimization and representation challenges. We identify encoder Jacobian conditioning as a key factor in trimodal contrastive learning: poorly conditioned encoders exhibit collapsing or amplified singular-value spectra, leading to exploding Jacobian condition numbers and degraded multimodal alignment. We introduce geometry-preserving encoders (GPEs) by directly conditioning the Jacobian through regularization and demonstrating that simple modifications like LeakyReLU activations and residual paths recover these geometric benefits. Across a synthetic benchmark and four real-world datasets including missing modalities, improving Jacobian conditioning boosts retrieval and linear probe performance across multiple contrastive objectives, whereas expressive objectives yield little benefit in linear probes. More broadly, our results show that multimodal contrastive learning depends not only on objective expressivity, but also on the geometric and optimization properties of the underlying encoders.

Problem

• multimodal contrastive learning
• Jacobian conditioning
• geometry preservation
• encoder representation
• higher-order alignment

Innovation

• geometry-preserving encoders
• Jacobian conditioning
• multimodal contrastive learning
• LeakyReLU
• residual paths