GRACE: Generative Recommender Acceleration Engine for Real-Time Ads Retrieval

2026-08-01

AI Summary

This work addresses the dual challenges of stringent eligibility verification and high computational latency/cost in generative recommender systems for high-concurrency real-time ad retrieval by proposing the GRACE system. Its core innovations include a Generative Target Matching (GTM) mechanism leveraging semantic ID (SID) prefix constraints to substantially improve ad targeting rule compliance, and a streamlined encoder-decoder Transformer architecture incorporating bitmask and Bloom filter matching, a customized attention kernel, KV caching, and wide-beam decoding optimizations to drastically reduce computational overhead. Experimental results demonstrate that GRACE increases ad matching pass rates from 23.55% to 40.42%, reduces cross-attention and self-attention latencies by 68.0× and 23.4–25.8× respectively on GH200 hardware, and achieves an overall decoding speedup of 11.1×.

Abstract

Productionizing generative recommenders for high-volume, real-time ads retrieval creates two serving challenges: eligibility, ensuring that each generated ad is eligible for the request under the advertiser's audience targeting rules, and compute, which requires meeting strict latency and GPU cost requirements while remaining capable of generating thousands of ads per request with wide-beam decoding. This paper presents GRACE, a serving system for ads generative retrieval that addresses both challenges. For eligibility, GRACE introduces Generative Target Matching (GTM), which extends catalog-valid constrained decoding with personalized filtering over Semantic ID (SID) prefixes using bitmask and Bloom filter matchers derived from targeting rules. SID-level GTM improves final ad-level target matching pass rate from 23.55% to 40.42% over constrained decoding alone. For compute-cost and latency, GRACE targets encoder-decoder Transformers, which are more lightweight than LLMs. It redesigns the decoder around the wide-beam, short-sequence regime, covering attention kernels, KV cache, and beam search optimizations. On NVIDIA GH200, compared with the faster of FlashAttention-2 and FlashAttention-3 baselines, GRACE improves cross-attention latency by 68.0 times and self-attention latency by 23.4-25.8 times across decode steps. Together, these changes reduce decoder latency by 11.1 times, keeping ads generative retrieval within latency and compute requirements.

Problem

• generative recommenders
• real-time ads retrieval
• eligibility
• compute efficiency
• latency constraints

Innovation

• Generative Target Matching
• Semantic ID
• Constrained Decoding
• Encoder-Decoder Transformer
• Wide-Beam Decoding