Gramian Multimodal Representation Learning and Alignment

2024-12-16

arXiv.org

AI Summary

To address the challenge of joint alignment across three or more modalities, this paper proposes a global geometric alignment framework that abandons conventional pairwise contrastive learning. The core innovation is the Gramian Alignment Measure (GRAM), which achieves synchronous, anchor-free alignment of *n* modalities in high-dimensional embedding space by minimizing the Gram determinant volume of the parallelotope spanned by their embeddings. The proposed contrastive loss—based on the volume of the Gram matrix—is agnostic to the number of modalities and serves as a drop-in replacement for cosine similarity. Evaluated on video-audio-text cross-modal retrieval and audio-video classification tasks, our method establishes new state-of-the-art performance, demonstrating both the effectiveness and generalizability of the geometric alignment paradigm.

Technology Category

• Machine Learning: Multimodal Learning
• Computer Vision: Multi-modal Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs

Abstract

Human perception integrates multiple modalities, such as vision, hearing, and language, into a unified understanding of the surrounding reality. While recent multimodal models have achieved significant progress by aligning pairs of modalities via contrastive learning, their solutions are unsuitable when scaling to multiple modalities. These models typically align each modality to a designated anchor without ensuring the alignment of all modalities with each other, leading to suboptimal performance in tasks requiring a joint understanding of multiple modalities. In this paper, we structurally rethink the pairwise conventional approach to multimodal learning and we present the novel Gramian Representation Alignment Measure (GRAM), which overcomes the above-mentioned limitations. GRAM learns and then aligns $n$ modalities directly in the higher-dimensional space in which modality embeddings lie by minimizing the Gramian volume of the $k$-dimensional parallelotope spanned by the modality vectors, ensuring the geometric alignment of all modalities simultaneously. GRAM can replace cosine similarity in any downstream method, holding for 2 to $n$ modalities and providing more meaningful alignment with respect to previous similarity measures. The novel GRAM-based contrastive loss function enhances the alignment of multimodal models in the higher-dimensional embedding space, leading to new state-of-the-art performance in downstream tasks such as video-audio-text retrieval and audio-video classification. The project page, the code, and the pretrained models are available at https://ispamm.github.io/GRAM/.

Problem

• Aligns multiple modalities for unified understanding
• Replaces cosine similarity with GRAM measure
• Enhances multimodal model alignment in embedding space

Innovation

• Gramian Representation Alignment Measure
• Higher-dimensional space alignment
• Contrastive loss function enhancement