Long-Term Optimization for Large-Scale Generative Retrieval with Off-Policy REINFORCE

2026-07-02

AI Summary

This work addresses the limitation of existing generative retrieval methods in optimizing long-term user satisfaction, as they predominantly rely on supervised next-item prediction. The authors formulate recommendation as a session-level sequential decision-making problem and train an autoregressive generative retriever using off-policy REINFORCE with pre-collected data. A user feedback model is incorporated to enable offline evaluation and optimize long-term rewards at test time. Key contributions include a multi-step importance weight approximation tailored for recommendation, a doubly robust off-policy evaluation method, and a test-time strategy that simulates future user responses via the feedback model to guide recommendation selection. Evaluated on the Yambda-5B dataset, the approach significantly improves offline estimates of cumulative session-level rewards while preserving retrieval quality, with further gains achieved during inference through future response simulation.

Abstract

Generative retrieval has become a popular paradigm for large-scale recommendation. However, it is typically trained with supervised next-item prediction objectives that do not directly optimize long-term user satisfaction. In this work, we formulate recommendation as a session-level sequential decision-making problem and introduce an autoregressive approach for training generative retrievers with off-policy REINFORCE on pre-collected data. Unlike the one-step off-policy correction used in prior work, we propose a multi-step approximation of importance weights enabled by the autoregressive formulation. To support offline evaluation, we train a user feedback model that simulates user responses to generated recommendations. This lets us adapt doubly robust off-policy evaluation for sequential decision-making to recommendation, a setting that has received limited attention. We further introduce a feedback-model-based test-time scaling procedure that simulates future responses and selects recommendations with the highest predicted long-term returns. Experiments on the public large-scale Yambda-5B dataset show that our RL agent improves offline estimates of cumulative session reward over next-item and next-positive prediction baselines, while largely preserving retrieval quality. Moreover, allocating more inference-time compute to simulating future responses improves model-based long-term return estimates without updating the policy.

Problem

• generative retrieval
• long-term user satisfaction
• sequential decision-making
• off-policy reinforcement learning
• recommendation

Innovation

• generative retrieval
• off-policy REINFORCE
• multi-step importance weighting
• user feedback model
• long-term reward optimization