Exponential Reward Weighting for Fine-Tuning Generative Recommenders under Sparse and Noisy Feedback

2026-08-01

AI Summary

This work addresses the challenges of fine-tuning generative recommender systems under sparse and noisy user feedback, where direct optimization of unreliable reward signals often leads to overfitting. To mitigate this issue, the authors propose Exponential Reward-weighted Supervised Fine-Tuning (Exp-RSFT), which introduces a temperature parameter into the reward-weighting mechanism. By exponentially weighting logged interactions as exp(r/λ), the method directly optimizes the original reward signal without requiring online exploration or human preference data. Theoretical analysis reveals a trade-off between coverage cost and noise cost, with the temperature parameter enabling an optimal balance. Extensive experiments on three public benchmarks and one industrial-scale dataset demonstrate that model performance exhibits an inverted U-shaped relationship with temperature, consistently outperforming PPO and DPO while yielding stable improvements in ranking effectiveness.

Abstract

In recommendation systems, users interact with only a small fraction of a vast item catalog, producing feedback that is both sparse and noisy. This challenges post-training generative recommenders: reward models trained from logged interactions often fail to generalize, while directly optimizing imperfect rewards can lead to reward over-optimization. We propose Exponential reward-weighted fine-tuning (Exp-RSFT), where each logged interaction is weighted by $\exp(r/λ)$, avoids this failure by optimizing directly on the logged rewards, with the temperature $λ$ regularizing against their noise. We theoretically show that Exp-RSFT's suboptimality decomposes into two costs: a coverage cost arising from limitations of the logging policy and a noise cost from imperfect feedback. The temperature $λ$ balances these competing effects, yielding an optimal tradeoff between exploiting high-reward behavior and robustness to noise. Across three public benchmarks and a large-scale industrial dataset, we verify this theoretical prediction: performance follows an inverted-U trend as a function of $λ$, while PPO and DPO often over-optimize unreliable reward models and degrade recommendation quality. Exp-RSFT consistently improves ranking performance without requiring online exploration or preference data.

Problem

• sparse feedback
• noisy feedback
• reward over-optimization
• generative recommenders
• recommendation systems

Innovation

• Exponential reward weighting
• Generative recommenders
• Sparse and noisy feedback
• Reward over-optimization
• Temperature regularization