Learning Relative Representations for Fine-Grained Multimodal Alignment with Limited Data

2026-05-16

AI Summary

This work addresses the limitation of existing posterior multimodal alignment methods, which rely solely on global representations and thus struggle to support fine-grained cross-modal tasks under paired data scarcity. To overcome this, the authors propose a novel posterior alignment approach based on relative representations, introducing for the first time a token-level relative representation mechanism into posterior alignment. By incorporating lightweight, learnable anchors within each modality’s embedding space, the method models similarity relationships between image and text tokens, enabling fine-grained structural alignment. Notably, it avoids complex projection layers and achieves effective cross-modal fine-grained correspondence through anchor optimization alone. Extensive experiments demonstrate significant performance gains over state-of-the-art methods on zero-shot classification, cross-modal retrieval, and zero-shot segmentation, validating the effectiveness and generalizability of the proposed fine-grained alignment strategy.

Abstract

Multimodal pre-training demonstrates strong generalization performance, but this paradigm is often impractical in domains where paired data are scarce. A promising alternative is post-hoc multimodal alignment, which aligns separately pre-trained unimodal encoders using a limited number of paired examples. However, existing methods focus primarily on aligning global representations, missing patch-token relations. This may hinder transfer to tasks that require fine-grained cross-modal matching beyond coarse sample-level semantics. To address this issue, we propose a post-hoc alignment method that learns token-level cross-modal structure using relative representations. Specifically, we represent images and texts through their token-level similarities to a set of learnable anchors in each modality space, which are trained to induce consistent cross-modal similarity patterns for matched pairs. Despite learning only the anchors without heavy projection layers, our approach consistently outperforms existing methods in zero-shot classification, cross-modal retrieval, and zero-shot segmentation by a substantial margin. This highlights the importance of modeling fine-grained cross-modal structure for effective post-hoc multimodal alignment with limited paired data.

Problem

• fine-grained alignment
• multimodal learning
• limited paired data
• post-hoc alignment
• token-level relations

Innovation

• relative representations
• fine-grained alignment
• post-hoc multimodal alignment
• token-level cross-modal structure
• limited paired data