Douyin Multimodal Embedding Model Technical Report

2026-08-03

AI Summary

This work addresses the challenge of simultaneously achieving efficient retrieval and fine-grained semantic matching in billion-scale multimodal platforms by proposing DME, a two-stage trained multimodal embedding model. In the first stage, a unified embedding space is established through large-scale contrastive learning. The second stage innovatively integrates Evidence-Grounded Typed Latent Reasoning with Cross-Conditional Reconstruction to enhance semantic representation without increasing online inference overhead. Evaluated on MMEB-v2, DME achieves scores of 74.8 and 78.4 on the 2B and 9B versions, respectively, substantially outperforming baseline methods. Internal evaluations at TikTok demonstrate a 2.92% improvement, and A/B testing reveals a 0.1% increase in lifetime value, leading to deployment across core applications including generative modeling, image search, and AI-powered search.

Abstract

Multimodal representation learning is a cornerstone of modern AI. By encoding multimodal queries and targets into vectors, it powers industrial search and recommendation and underpins modern agents. Real-world platforms with complex modalities and massive-scale content, such as Douyin, Xiaohongshu, and YouTube, demand both efficiency under billion-scale indexing and fine-grained discrimination for hard matching. Existing MLLM embedding models rarely satisfy both. Contrastive models are efficient but rely on pair-level supervision too coarse for fine-grained distinctions, while CoT-based models improve discrimination through explicit generation impractical to serve online. We present Douyin Multimodal Embedding (DME), a model trained in two stages to combine both strengths. Stage 1 performs large-scale contrastive pre-training that establishes a unified multimodal embedding space with broad modality and task coverage. Stage 2 supplements semantic sufficiency, the property that an embedding is grounded in retrieval-relevant evidence and preserves fine-grained counterpart-side semantics, via two mechanisms. Evidence-Grounded Typed Latent Reasoning organizes retrieval evidence through hidden-space latent reasoning, and Cross-Conditional Reconstruction enforces counterpart-side semantics through cross-directional autoregressive reconstruction. Both act only during training and add only marginal query-side overhead, so DME serves as efficiently as a standard contrastive encoder. On MMEB-v2, DME reaches state-of-the-art results at comparable scales for its 2B and 9B variants (74.8 and 78.4), with especially strong video and visual-document tasks. In production, DME delivers a 2.92% relative gain on Douyin's in-house offline evaluation set, is deployed across Douyin scenarios such as generative, image, and AI search, and yields a 0.1% Lifetime (LT) gain in online A/B testing on Douyin search.

Problem

• multimodal embedding
• fine-grained discrimination
• billion-scale indexing
• retrieval
• semantic representation

Innovation

• multimodal embedding
• semantic sufficiency
• latent reasoning
• cross-conditional reconstruction
• contrastive pre-training