Flux-OPD: On-Policy Distillation with Evolving Contexts

2026-07-30

AI Summary

This work addresses the challenge of lacking verifiable rewards in open-domain large language model training, which hinders the conversion of task preferences into effective supervision signals. To overcome this, the authors propose an evolution-aware contextual policy distillation method that dynamically captures task preferences through evolving contexts and leverages contextual discrepancies to generate corrective signals. The study elucidates the role of context in reverse KL–based distillation objectives and introduces a conflict term as an adaptive weight modulating correction strength. By integrating a context-conditional teacher model, a geometric mean distillation objective, and a conflict-aware weighting mechanism, the approach achieves stable and efficient knowledge transfer. Experiments demonstrate that the proposed method significantly outperforms existing policy distillation techniques on open-domain tasks, validating the efficacy of combining teacher supervision with evolutionary contextual signals.

Abstract

Large language model training in open-ended domains lacks verifiable rewards, making task preferences difficult to formalize as effective supervision. Contexts can convey such preferences, yet provide little additional supervision once distilled into the student, motivating contexts that evolve with student performance. However, directly using evolving contexts as in-training supervision results in an unstable distillation target and conflicting distributions, requiring mechanisms to stabilize target and downweight conflicts. In this paper, we analyze the effect of contexts through a decomposition of the reverse KL objective, revealing two findings: the student is distilled toward the geometric mean of context-conditioned teachers, and the objective contains a conflict term that measures conflicts among these teachers. Based on this decomposition, we propose Flux-OPD, an OPD paradigm that uses evolving contexts as in-training supervision to capture task preferences in open-ended domains. Flux-OPD treats the differences between context-conditioned and context-free teachers as contextual difference signals, injects them as contextual corrections into the context-free teacher anchor, and weights their correction strength using the conflict term as an indicator. Experiments on open-ended tasks show that Flux-OPD outperforms existing OPD paradigms, highlighting the potential to combine teacher supervision with evolving contexts.

Problem

• open-ended domains
• task preferences
• on-policy distillation
• evolving contexts
• distillation stability

Innovation

• On-Policy Distillation
• Evolving Contexts
• Reverse KL Decomposition
• Contextual Correction
• Teacher Conflict