Binding Multiple Modalities via Multimodal Wasserstein Barycenter

2026-09-27

AI Summary

This study addresses the challenge of constructing a balanced representation space in multimodal learning by proposing BaryBind, a novel framework that transports modality-specific features toward their Wasserstein barycenter to establish a unified semantic space. To this end, the method introduces a volume alignment objective that leverages the barycentric simplex volume as a global similarity measure, combined with geometric projection and simplex construction techniques for efficient optimization. Experimental results demonstrate that BaryBind achieves superior performance on multimodal retrieval and classification tasks while exhibiting notable robustness against missing modalities and strong scalability.

Abstract

Multimodal learning beyond two modalities commonly leverages a specific modality (e.g., text) to bind other modalities. However, how to establish a more balanced representation space that approximates shared semantics while respecting the holistic geometry of $n$-modal data remains challenging. In this work, we present BaryBind, which aims to transport the specific modality towards the Wasserstein barycenter (WB) optimized across all modalities and introduces a volumetric alignment objective to establish a unified semantic space around the WB embedding. Specifically, we project specific modalities to the WB, which minimizes the average Wasserstein distances to multimodal distributions and serves as the anchor for subsequent alignment. We then construct a barycenter simplex, whose volume is taken as a similarity metric for global alignment centered at the WB. Experiments show that BaryBind achieves competitive performance in text-video-audio retrieval, classification, videoQA, and cross-modal generation tasks, along with robustness under modality absence and scalability to more than three modalities. Code is released at https://github.com/xl-tang3/BaryBind.

Problem

• Multimodal Learning
• Wasserstein Barycenter
• Representation Space
• Shared Semantics
• Holistic Geometry

Innovation

• Multimodal Wasserstein Barycenter
• Volumetric Alignment
• Barycenter Simplex
• Optimal Transport
• Multimodal Learning