Beyond Dark Knowledge: Mixup-Based Distillation for Reliable Predictions

2026-06-10

AI Summary

This work addresses a critical yet overlooked issue in knowledge distillation when combined with mixup: the teacher model is queried on out-of-distribution neighborhoods, leading to supervision signals corrupted by distributional confusion and impairing knowledge transfer. The authors propose a distillation mechanism that applies mixup exclusively to the student, thereby revealing for the first time the distributional mismatch between teacher and student. They demonstrate that calibration capability can be transferred independently of accuracy. By integrating temperature scaling with calibration-aware evaluation, the method’s universality across teachers of varying capacities is validated on CIFAR and ImageNet. Results show that the student not only achieves significantly higher accuracy but also exhibits an order-of-magnitude reduction in overconfidence, along with improved uncertainty estimation and representation geometry.

Abstract

Knowledge Distillation (KD) and mixup have proven effective at inducing smoothness in class boundaries; KD captures inherent class relationships in probability distributions, and mixup enforces them through convex combinations of inputs. Their interaction, however, remains poorly understood, particularly when mixup is applied only during student training. In this setting, the teacher is queried on inputs drawn from a vicinal distribution it never saw during training, a controlled mismatch whose effect on knowledge transfer has not been characterised. We show that this mismatch causes the teacher's supervisory signal to be dominated by distributional confusion rather than inter-class structure. Despite it, the student does not merely imitate the teacher: it independently acquires greater linearity in the vicinal region, a structural property that the teacher lacks, and goes beyond dark-knowledge transfer. KD with mixup consistently improves student accuracy and reduces overconfidence by an order of magnitude relative to the baseline, across CIFAR and ImageNet with varying-capacity teachers. Crucially, calibration propagates from teacher to student independently of accuracy transfer, and temperature scaling governs a measurable accuracy-calibration trade-off that becomes more pronounced under vicinal training. These results reframe mixup distillation not as a degraded version of standard KD, but as a richer transfer channel that simultaneously shapes discriminative performance, uncertainty estimation, and representational geometry.

Problem

• Knowledge Distillation
• mixup
• distributional mismatch
• vicinal distribution
• calibration

Innovation

• mixup distillation
• knowledge distillation
• vicinal distribution
• model calibration
• representational geometry