Variable-Length Semantic IDs for Recommender Systems

2026-02-18

AI Summary

This work addresses the challenges posed by the vast item vocabulary and the inflexibility of fixed-length semantic IDs in recommendation systems, which struggle to accommodate the skewed item frequency distribution. To this end, the authors propose a variable-length semantic ID framework that, for the first time, adapts the variable-length coding paradigm from emergent communication to recommender systems. By leveraging a discrete variational autoencoder with Gumbel-Softmax reparameterization, the method probabilistically assigns shorter codes to high-frequency items and longer codes to long-tail items, enabling adaptive and compact representations. Compared to approaches relying on REINFORCE-based optimization, the proposed method offers more stable training, effectively mitigates the vocabulary gap, and significantly enhances the modeling capability and overall performance of generative recommender models—particularly for sparse, long-tail items.

Technology Category

• Data Mining & Knowledge Management: Recommender Systems
• Machine Learning: Deep Generative Models & Autoencoders
• Natural Language Processing: Generation

Application Category

• User Modeling, Personalization and Recommendation: Psychology-informed user models and recommender systems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs

Abstract

Generative models are increasingly used in recommender systems, both for modeling user behavior as event sequences and for integrating large language models into recommendation pipelines. A key challenge in this setting is the extremely large cardinality of item spaces, which makes training generative models difficult and introduces a vocabulary gap between natural language and item identifiers. Semantic identifiers (semantic IDs), which represent items as sequences of low-cardinality tokens, have recently emerged as an effective solution to this problem. However, existing approaches generate semantic identifiers of fixed length, assigning the same description length to all items. This is inefficient, misaligned with natural language, and ignores the highly skewed frequency structure of real-world catalogs, where popular items and rare long-tail items exhibit fundamentally different information requirements. In parallel, the emergent communication literature studies how agents develop discrete communication protocols, often producing variable-length messages in which frequent concepts receive shorter descriptions. Despite the conceptual similarity, these ideas have not been systematically adopted in recommender systems. In this work, we bridge recommender systems and emergent communication by introducing variable-length semantic identifiers for recommendation. We propose a discrete variational autoencoder with Gumbel-Softmax reparameterization that learns item representations of adaptive length under a principled probabilistic framework, avoiding the instability of REINFORCE-based training and the fixed-length constraints of prior semantic ID methods.

Problem

• semantic IDs
• recommender systems
• variable-length representation
• item cardinality
• vocabulary gap

Innovation

• variable-length semantic IDs
• discrete variational autoencoder
• Gumbel-Softmax reparameterization
• recommender systems
• emergent communication