Scaling Recommender Transformers to One Billion Parameters

2025-07-21

AI Summary

To address scalability bottlenecks of large language models in recommender systems, this paper proposes a billion-parameter-scale Transformer-based recommendation architecture. The core methodological innovation lies in reformulating recommendation as a sequence transduction task and introducing a dual-path autoregressive learning paradigm: jointly modeling implicit feedback prediction (e.g., click/skip) and next-item prediction, thereby decoupling user intent modeling from sequential item generation. The model is trained end-to-end on long user behavioral sequences. Online A/B testing demonstrates statistically significant improvements: a 2.26% increase in total playback duration and a 6.37% uplift in like probability—setting a new record for performance gains among deep learning–based recommenders on the platform. These results underscore substantial advances in training efficiency and generalization capability for ultra-large-scale recommendation models.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Search and Optimization: Learning to Search

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

While large transformer models have been successfully used in many real-world applications such as natural language processing, computer vision, and speech processing, scaling transformers for recommender systems remains a challenging problem. Recently, Generative Recommenders framework was proposed to scale beyond typical Deep Learning Recommendation Models (DLRMs). Reformulation of recommendation as sequential transduction task led to improvement of scaling properties in terms of compute. Nevertheless, the largest encoder configuration reported by the HSTU authors amounts only to ~176 million parameters, which is considerably smaller than the hundreds of billions or even trillions of parameters common in modern language models. In this work, we present a recipe for training large transformer recommenders with up to a billion parameters. We show that autoregressive learning on user histories naturally decomposes into two subtasks, feedback prediction and next-item prediction, and demonstrate that such a decomposition scales effectively across a wide range of transformer sizes. Furthermore, we report a successful deployment of our proposed architecture on a large-scale music platform serving millions of users. According to our online A/B tests, this new model increases total listening time by +2.26% and raises the likelihood of user likes by +6.37%, constituting (to our knowledge) the largest improvement in recommendation quality reported for any deep learning-based system in the platform's history.

Problem

• Scaling transformer models for recommender systems beyond typical DLRMs
• Training large transformer recommenders with up to a billion parameters
• Improving recommendation quality via autoregressive learning on user histories

Innovation

• Scaling transformers to one billion parameters
• Autoregressive learning on user histories
• Deployment on large-scale music platform