Improving Relative Representations with Learned Anchors and Whitened Inner Products

2026-05-28

AI Summary

This work addresses the challenge of constructing modular AI systems from independently trained neural models, whose latent spaces are often incompatible. To overcome this, the authors propose an enhanced relative representation framework that leverages learnable semantic prototypes as cross-model anchors and replaces cosine similarity with a whitened inner product that preserves magnitude information while being invariant to affine transformations, thereby enabling geometry-aware similarity measurement. This approach effectively aligns embedding spaces across heterogeneous architectures—including small language models—and achieves near-lossless information transfer and stable zero-shot communication in vision-and-language tasks, significantly improving cross-model representational consistency.

Abstract

Independently trained neural models typically converge to incompatible latent representations, creating a fundamental barrier to highly modular AI systems. While Relative Representations (RR) address this by mapping absolute coordinates to a shared space defined by similarities to common anchor points, traditional implementations rely on randomly sampled anchors and cosine similarity, which frequently fail to capture the anisotropic geometries of modern architectures like Transformers. In this work, we propose a robust framework for cross-model communication based on two improvements. We learn anchors as robust semantic prototypes and utilize a geometry-aware similarity metric which preserves discriminative magnitude information and is invariant to affine shifts. Our approach demonstrates significant gains in performance and consistency across vision and language tasks. Notably, it enables nearly lossless information transfer and stable zero-shot communication even between highly heterogeneous architectures, such as small language models of varying scales.

Problem

• Relative Representations
• latent representations
• cross-model communication
• anisotropic geometry
• modular AI systems

Innovation

• Learned Anchors
• Whitened Inner Products
• Relative Representations
• Cross-model Communication
• Geometry-aware Similarity