CADENCE: Closing the Reasoning Gap via Coverage-Adaptive On-Policy Distillation

2026-07-18

AI Summary

Existing policy-based knowledge distillation methods struggle to effectively transfer reasoning capabilities from large language models due to cold-start collapse, state-agnostic divergence scheduling, and sparse binary rewards. This work proposes CADENCE, a novel framework incorporating the DRIFT mechanism and six key components—COVA, FTB, CCD, LAP, EMR, and BSD—to enable coverage-aware distillation, gradient focusing, and exploitation of partially correct feedback. CADENCE achieves this through token-level forward–backward KL dynamic mixing, coverage-adaptive β scheduling, dense numerical proximity rewards, entropy-matching regularization, and reinforcement learning biased toward solution conciseness. Evaluated on GSM8K, a 0.5B student model trained with CADENCE improves accuracy from 48.7% to 69.8%, closing 63.2% and 76.2% of the reasoning gaps relative to 1.5B and 3B teacher models, respectively, while requiring only a single Mac Studio for efficient training.

Abstract

On-policy knowledge distillation transfers reasoning from large teachers to compact students, but existing approaches suffer three compounding failure modes: (i) cold-start collapse, where a fresh student assigns near-zero mass to teacher-preferred tokens; (ii) state-agnostic divergence scheduling, where time-only forward/reverse-KL interpolation ignores the student's coverage state; and (iii) binary reward sparsity, where pass/fail signals discard information from partially correct traces. We present CADENCE, a unified framework with a targeted fix for each. Its DRIFT mechanism schedules a per-token convex mixture of forward-KL and reverse-KL surrogate objectives on student-sampled trajectories (per-token surrogates, not sequence-level KL gradient estimators). Six components extend it: (A) COVA, a coverage-adaptive $β$ schedule accelerating the forward-to-reverse transition; (B) FTB, a forking-token boost concentrating gradient at high-entropy positions via a globally-normalized entropy reference; (C) CCD, a dense reward adding numerical-proximity partial credit for incorrect-but-close traces; (D) LAP, brevity-preferential correct-rollout reinforcement; (E) EMR, an entropy-matching calibration regularizer; (F) BSD, a bootstrapped self-distillation phase. On GSM8K and MATH-500 (corrected 512-token protocol, 5 seeds, reported std), CADENCE distills a 0.5B student from a 1.5B teacher to 69.8 $\pm$ 0.5% GSM8K pass@1 (from 48.7% pretrained; 63.2% of the teacher gap closed) and to 72.1 $\pm$ 0.4% with a 3B teacher (76.2% closed), beating the strongest matched-compute label-using baseline (DRIFT+binary reward) by +4.4 $\pm$ 0.7 points. All experiments run on a single Apple Mac Studio (M-series, 64GB unified memory), showing principled distillation reaches strong reasoning quality without datacenter-scale hardware.

Problem

• on-policy distillation
• reasoning gap
• cold-start collapse
• reward sparsity
• coverage-adaptive scheduling

Innovation

• on-policy distillation
• coverage-adaptive scheduling
• dense reward
• KL divergence mixing
• reasoning compression