FlowCTS: On-policy Continuous Trajectory Supervision of Flow Models

2026-07-27

AI Summary

This work addresses the challenges in existing on-policy distillation based on KL divergence (KL-OPD) for flow models, which suffers from temporal supervision misalignment due to auxiliary SDE transition kernels and is further hindered by sparse rewards and exposure bias. To overcome these limitations, the authors propose FlowCTS, a novel approach that introduces continuous trajectory supervision into the on-policy distillation framework for flow models. By continuously aligning student and reference trajectories originating from the same state, FlowCTS leverages the integral relationship between trajectories and the velocity field to construct a time-weighted upper bound for velocity matching, which is then discretized into an optimizable objective. This method effectively mitigates temporal supervision mismatch and accommodates multiple reference trajectories, achieving significant improvements over KL-OPD and hybrid reward reinforcement learning baselines on GenEval (0.90→0.93), OCR (0.90→0.92), and PickScore (22.75→23.06).

Abstract

While on-policy distillation (OPD) effectively addresses sparse rewards and exposure bias in large language model post-training, its extension to flow models remains underexplored. To this end, we propose Flow Continuous Trajectory Supervision (FlowCTS), which matches subsequent student and reference trajectories initialized from the same student-visited state. Using the integral relation between trajectories and velocity fields, we derive a temporally weighted velocity-matching upper bound and discretize it into practical objectives parameterized by the number of supervision steps. Under a multi-reference setup, single-state FlowCTS-OPD outperforms vanilla KL-based OPD with faster convergence. FlowCTS-OPD improves GenEval from 0.90 to 0.93, OCR from 0.90 to 0.92, and PickScore from 22.75 to 23.06, while outperforming a mixed-reward RL baseline across all target metrics. Further analysis reveals a clear temporal supervision mismatch in vanilla KL-based OPD arising from its auxiliary SDE transition kernels. Beyond on-policy setting,FlowCTS also consistently outperforms vanilla SFT , particularly on OCR, while increasing supervision steps exhibit a trade-off between richer trajectory information and greater optimization difficulty.

Problem

• flow models
• on-policy distillation
• trajectory supervision
• exposure bias
• sparse rewards

Innovation

• FlowCTS
• on-policy distillation
• continuous trajectory supervision
• velocity matching
• flow models