Distill What the Student Can See: Fisher-Projected On-Policy Distillation for Vision-Language Models

2026-08-02

AI Summary

This work addresses a key limitation in existing vision-language model distillation methods, which assume that the teacher’s output distribution is fully suitable as a learning target for the student, thereby overlooking the student’s capacity constraints in perceiving or expressing certain visual details. To remedy this, the authors propose Fisher-Projected On-Policy Distillation (FP-OPD), which introduces, for the first time, a local realizability constraint grounded in Fisher information. Specifically, FP-OPD estimates the local tangent space of the student model with respect to input visual features and projects the log-probability discrepancy between teacher and student onto this space, thereby transmitting only those correction signals realizable within the student’s representational capacity. Combining on-policy trajectory sampling, continuous visual perturbations, Fisher matrix projection, and full-vocabulary reverse KL optimization, FP-OPD achieves consistent gains across all seven vision-language benchmarks under an 8B→2B distillation setting, improving average scores by 2.77 points over the pretrained student and by 1.60 points over standard OPD.

Abstract

On-policy distillation (OPD) samples trajectories from the current student policy and minimizes token-level divergence between student and teacher next-token distributions at prefixes along those trajectories. This aligns the distillation states with the student's own generation distribution. However, it still assumes that the complete teacher distribution is an appropriate target across student capacities. In vision--language reasoning, teacher corrections can depend on visual distinctions that a compact student cannot represent. Our target-scaling study shows that, as the target approaches the complete teacher distribution, the student realizes less of the prescribed shift and obtains worse downstream performance. We therefore propose \emph{Fisher-Projected On-Policy Distillation} (FP-OPD), which distills only locally realizable teacher corrections. FP-OPD uses continuous visual perturbations to estimate the student's local visual tangent space and projects the centered teacher--student log-probability gap onto this space under the student's Fisher metric. The resulting capacity-aware target is optimized with full-vocabulary reverse KL on student trajectories, retaining the standard OPD framework. In 8B-to-2B distillation, FP-OPD improves all seven evaluated multimodal benchmarks. It raises the average score by 2.77 points over the pretrained student and by 1.60 points over standard OPD. These results demonstrate that locally realizable teacher corrections provide a more effective target for distilling compact vision--language models.

Problem

• on-policy distillation
• vision-language models
• model capacity
• teacher-student mismatch
• knowledge distillation

Innovation

• Fisher-Projected On-Policy Distillation
• vision-language models
• knowledge distillation
• capacity-aware targets
• local visual tangent space