Hierarchical Residual Policy Optimization for Generative Recommendations

2026-08-01

AI Summary

This work addresses the challenge of sparse, high-variance, and hierarchically dependent token-level credit assignment in generative recommender systems during post-training, where user feedback is only observed on the final exposed item, making it difficult to effectively optimize individual tokens of semantic identifiers (SIDs). To tackle this, the authors propose Hierarchical Residual Policy Optimization (HRPO), a novel framework that integrates hierarchical credit assignment with residual learning. HRPO estimates the utility of SID prefixes, decomposes residual credits, and accumulates them into “pending credits” to produce dense, token-aligned learning signals. The method further incorporates group reward smoothing, clipped updates, group-normalized advantage estimation, and KL regularization to enable conservative yet efficient per-token policy improvement. Experiments demonstrate that HRPO significantly enhances session-level utility and key business metrics on both public benchmarks and large-scale commercial A/B tests.

Abstract

Generative recommenders select items by autoregressively decoding semantic identifiers (SIDs), whose token positions induce a coarse-to-fine hierarchy over the item space. In practice, SID decoders are trained via supervised next-token prediction, which imitates logged trajectories rather than directly optimizing downstream utility. This motivates post-training with outcome feedback to guide decoding toward higher utility. However, logged feedback is only observed for the final exposed item, causing most post-training methods to operate at the item level and broadcast the same terminal signal across all SID tokens. As a result, token-level credit assignment becomes sparse, high-variance, and layer-dependent. To this end, we propose Hierarchical Residual Policy Optimization (HRPO), a post-training framework that converts item-level outcomes into dense, token-aligned learning signals for conservative token-wise improvement. Specifically, HRPO first estimates SID prefix-level utilities via group-wise reward smoothing over feature-based user clusters. It then decomposes these utilities into residual token credits and accumulates them into credit-to-go signals. Finally, Residual-Return Policy Optimization (RRPO) optimizes the residual credits using clipped updates, group-normalized advantages, and KL regularization to preserve stability. Experiments on a public dataset and an online A/B test in a large-scale commercial system show consistent gains in session-level utility and key business metrics. Source code and the archived artifact are available for reproduction.

Problem

• generative recommendations
• credit assignment
• hierarchical token decoding
• post-training
• outcome feedback

Innovation

• Hierarchical Residual Policy Optimization
• Generative Recommendation
• Token-level Credit Assignment
• Reward Smoothing
• Residual-Return Policy Optimization