ROCS: Request-Oriented Compute Sharing for Efficient Large-Scale Recommendation

2026-07-30

AI Summary

This work addresses the high inference cost and scalability bottlenecks in large-scale recommendation systems, which persist despite advances in prediction accuracy. The authors propose a request-oriented computation sharing paradigm that defers interaction between user requests and candidate items and executes request-specific computations only once, substantially reducing inference overhead. Key innovations include Generalized Layer Masking (GLM), Deep Cross-Attention (DCA), and In-Kernel Broadcast Optimization (IKBO), which collectively enable isolation and efficient sharing of candidate representations during feature interaction and sequential modeling. Evaluated in a real-world production system, the approach achieves up to a 3× increase in queries per second (QPS) and a 50% throughput gain in typical scenarios, all while maintaining or even improving model quality—evidenced by a 0.5% reduction in LogLoss for short-video ranking.

Abstract

Modern recommendation models gain prediction quality by scaling feature-interaction and sequence modules, but production cost constraints cap how far systems can scale. In this work, we propose Request-Oriented Compute Sharing (ROCS), a modeling and inference paradigm that exploits a unique property of recommendation inference: each user request is evaluated against many candidates, while request-side features are shared across candidates. ROCS defers request-candidate interactions as late as possible, isolates candidate-dependent representations, and evaluates substantial portions of the model once per request rather than once per candidate, significantly improving inference efficiency while maintaining or improving prediction quality. To realize this paradigm, we develop Generalized Layer Masking (GLM) to enforce candidate isolation in feature-interaction architectures, and Deep Cross Attention (DCA) to extend request-oriented sharing to sequence architectures. To support efficient GPU deployment, we co-design In-Kernel Broadcast Optimization (IKBO) that significantly accelerates ROCS model execution. Experiments on public benchmarks show that ROCS consistently improves the quality-efficiency tradeoff across recommendation backbones. On production-scale workloads, ROCS achieves up to a 3x QPS improvement on retrieval models without quality degradation and a 0.5% relative LogLoss improvement with a 50% QPS gain on a short-form video ranking model. ROCS has been deployed across large-scale recommendation systems spanning ads and organic surfaces, retrieval and ranking stages, and more than two orders of magnitude in inference complexity, delivering significant online gains at reduced infrastructure cost.

Problem

• recommendation systems
• inference efficiency
• large-scale
• compute sharing
• quality-efficiency tradeoff

Innovation

• Request-Oriented Compute Sharing
• Generalized Layer Masking
• Deep Cross Attention
• In-Kernel Broadcast Optimization
• Efficient Recommendation Inference