SAF-OPD: Stable Advantage Fusion for On-Policy Distillation

2026-07-31

AI Summary

This work addresses the entropy collapse induced by magnitude and temporal mismatches in advantage signals when integrating reinforcement learning value regression (RLVR) with online policy distillation (OPD), which hinders exploration and performance gains. To mitigate these dual mismatches, we propose the Stable Advantage Fusion (SAF) framework, featuring a lightweight four-stage pipeline that incorporates sparsely compressed advantage scaling for magnitude control and a warm-up annealing schedule for temporal alignment—both independently tunable. Built upon the GRPO training framework with dynamic coefficient scheduling, SAF demonstrates consistent improvements across seven mathematical reasoning and code generation tasks on Qwen3-1.7B/4B/8B models, achieving aggregate score gains of 0.51–2.70%, enhanced training stability, and significant superiority over fixed-coefficient fusion baselines.

Abstract

Reinforcement learning with verifiable rewards (RLVR) broadcasts a single response-level reward to every token, while on-policy distillation (OPD) scores each token against a stronger teacher for a dense advantage but caps performance at teacher quality and discourages exploration beyond it. Their complementarity makes combining RLVR and OPD promising, but we find that fusing the two advantages with a fixed coefficient triggers entropy collapse from two miscalibrations: a magnitude mismatch, where token-level OPD advantages can spike far beyond the bounded RLVR advantage and erase its signal, and a temporal mismatch, where sustained full-strength OPD keeps pulling the student toward the teacher and limits exploration needed to surpass it. We propose SAF, a Stable Advantage Fusion framework that resolves both issues via a lightweight, four-stage pipeline applied only to the OPD advantage: a sparsify-then-compress mechanism for magnitude control paired with a warm-up-then-anneal mechanism for temporal control, with each stage independently switchable and adding negligible overhead. Instantiating RLVR with GRPO, we evaluate SAF across seven mathematical reasoning and code generation benchmarks with Qwen3-1.7B/4B/8B: SAF avoids entropy collapse and consistently outperforms fixed-coefficient GRPO+OPD fusion, improving the aggregate score by 0.51-2.70% across all six model-domain settings while achieving more stable training.

Problem

• on-policy distillation
• reinforcement learning with verifiable rewards
• advantage fusion
• entropy collapse
• token-level advantage

Innovation

• Stable Advantage Fusion
• On-Policy Distillation
• Entropy Collapse
• Advantage Calibration
• Reinforcement Learning with Verifiable Rewards