Tokens are All You Need: Dual-purpose Semantic IDs for Achieving LLM-Level I/O Efficiency in recommendation systems

2026-07-26

AI Summary

This work addresses the high memory overhead and inefficient I/O of dense embedding tables in large-scale recommender systems. The authors propose a dual-purpose semantic ID framework that, for the first time, leverages discrete tokens to jointly represent user/item identities in collaborative filtering and enable on-demand reconstruction of content embeddings. By integrating hierarchical quantization, learnable embedding tables, and a lightweight semantic decoder, the method eliminates the need for storing dense vectors while preserving strong representational capacity. This approach substantially reduces memory consumption and data footprint. The system has been successfully deployed in both retrieval and ranking modules of a major video platform, with offline and online experiments consistently demonstrating its superiority in both efficiency and recommendation performance.

Abstract

Large-scale recommendation systems face "Memory Wall" bottlenecks due to massive, dense embedding tables. While generative retrieval uses discrete tokens for IDs, high-dimensional context still relies on inefficient dense formats. Inspired by computer vision data compression, we propose Dual-purpose Semantic IDs to achieve LLM-level I/O efficiency. Our methodology uses hierarchical quantization to condense continuous embeddings into discrete Semantic IDs performing two concurrent roles: (1) Collaborative Identity: modeling user-item interactions via learnable embedding table; and (2) Content Reconstruction: using a lightweight Semantic Decoder for on-the-fly embedding approximation. This approach replaces massive vector storage with on-demand reconstruction, reducing system overhead and data footprints. We demonstrate the efficacy of our framework through offline evaluations and successful online deployment in production-scale ranking and retrieval systems at a major video sharing platform, showing that discrete tokens are indeed all you need for highly efficient, content-rich recommendation.

Problem

• Memory Wall
• embedding tables
• I/O efficiency
• recommendation systems
• dense representations

Innovation

• Semantic IDs
• Hierarchical Quantization
• Generative Retrieval
• Embedding Compression
• I/O Efficiency