ReLoop-UME: Recurrent Depth with Learnable Retrieval Registers for Universal Multimodal Embedding

2026-07-30

AI Summary

This work addresses the high latency and dependency on intermediate states inherent in existing general-purpose multimodal embedding methods, which often rely on sequential generation or explicit reasoning tokens. To overcome these limitations, the authors propose an efficient, low-latency multimodal embedding architecture structured into three stages: context encoding, retrieval feature formation, and embedding mapping. The design incorporates parameter-shared recurrent retrieval blocks and learnable retrieval registers, enabling computation to scale depth-wise within a fixed input token space without requiring additional reasoning tokens. This approach significantly enhances both efficiency and performance, outperforming current methods on the MMEB-V2 and MRMR benchmarks while achieving 44.9× faster inference than UME-R1 and 1.5× faster than PLUME.

Abstract

Universal multimodal embedding (UME) maps heterogeneous multimodal inputs into a shared embedding space. Existing UME models either form embeddings through single forward encoding or add computation through explicit rationale tokens and latent autoregressive states. Although token expansion can improve complex matching, serial generation increases retrieval latency and makes the final embedding depend on generated intermediate states. This raises a different question: can useful computation be expanded along model depth while keeping the token workspace fixed? We analyze positive-negative similarity separation at every layer of independently trained UME models and observe a shared progression: early layers contextualize multimodal inputs, a contiguous middle-to-late stage forms retrieval-discriminative features, and the final layers map them into the embedding space. Based on this finding, we propose ReLoop-UME, which executes the early layers once, recurrently reuses a parameter-shared retrieval-forming block, and applies the final mapping layers after the last loop. Learnable Retrieval Registers provide persistent retrieval-specific states that accumulate and exchange evidence across loops, with the final register serving as the embedding readout. On MMEB-V2 and MRMR, ReLoop-UME consistently improves retrieval across different backbones while running 44.9x faster than UME-R1 and 1.5x faster than PLUME.

Problem

• Universal Multimodal Embedding
• Retrieval Latency
• Embedding Space
• Multimodal Inputs
• Token Workspace

Innovation

• ReLoop-UME
• Universal Multimodal Embedding
• Learnable Retrieval Registers
• Recurrent Depth
• Fixed Token Workspace