Progressive Alignment of Recommender Foundation Model through Multi-Phase Post-Training

2026-08-07

AI Summary

This work addresses the challenge that existing recommendation foundation models struggle to align with real-world business metrics during fine-tuning, thereby limiting their practical effectiveness. To overcome this, the authors propose a three-stage progressive post-training framework: first, linear probing rapidly adapts the model to the downstream task; second, full-model fine-tuning refines overall performance; and third, reinforcement fine-tuning guided by a reward model leverages dense implicit feedback to optimize sparse business objectives. This approach decouples task adaptation from metric alignment by treating them in distinct phases—a strategy not previously explored. The method outperforms conventional single-stage fine-tuning in offline evaluations and demonstrates significant gains in online A/B tests, surpassing both standard fine-tuned foundation models and traditional non-foundation-model baselines in production recommendation quality.

Abstract

Foundation model(FM) for recommendation has shown strong ability to model long-horizon sequential user behavior. In practice, a single pretrained foundation model is often adapted to diverse downstream serving surfaces through Supervised Fine-Tuning(SFT). However, optimizing task-specific objectives such as clicks or likes does not necessarily align the serving policy with the business metrics that determine recommendation quality. We propose a three-phase progressive post-training framework that explicitly separates downstream adaptation from business-metric alignment. The adaptation stage is decomposed into Linear Probing(LP) and Full Fine-Tuning(FFT): LP first stabilizes randomly initialized downstream heads within a frozen pretrained representation space, and FFT then jointly specializes the full model for the target task. On top of this stabilized policy, Reinforcement Fine-Tuning(RFT) aligns the model with practical business objectives using a learned reward model. Rather than directly optimizing the serving policy on sparse business targets, we train the policy on dense implicit feedback and use business-metric supervision only for reward modeling. Offline experiments show that the progressive LP-FFT-RFT framework outperforms single-phase alternatives, and that reward-based alignment yields a stronger serving policy than directly using the reward model itself for ranking. Large-scale online A/B tests further show that the proposed framework improves production recommendation quality over a conventional non-foundation baseline. A reference implementation is available at https://github.com/webtoon/rec-fm-progressive-alignment

Problem

• Foundation Model
• Recommendation Alignment
• Business Metrics
• Supervised Fine-Tuning
• Reinforcement Learning

Innovation

• Progressive Alignment
• Recommender Foundation Model
• Reinforcement Fine-Tuning
• Linear Probing
• Reward Modeling