The Devil is in the Spectrum Bias: Spectrum-Balanced Feature Matching for Robust Representation Distillation

2026-09-28

AI Summary

Feature matching in traditional knowledge distillation tends to bias toward dominant spectral directions, neglecting low-variance yet task-relevant information. To address this limitation, this work proposes SpecMatch, an algorithm that for the first time reveals and rectifies the spectral bias inherent in L2 distance. By employing an adaptive weighting strategy, SpecMatch dynamically emphasizes under-optimized spectral directions, achieving spectrally balanced feature matching with minimal computational overhead and thereby improving the student model's reconstruction of teacher representations. Extensive evaluations across 42 vision configurations demonstrate that SpecMatch outperforms baseline methods in 40 settings, yielding significant performance gains in downstream tasks including classification, anomaly detection, and protein understanding. These results effectively overcome the limitations of conventional distillation approaches.

Abstract

Large visual foundation models have demonstrated remarkable transferability across a wide range of downstream tasks. To deploy such models efficiently, feature matching has become a popular knowledge distillation approach that transfers teacher representations to smaller student models without requiring labeled data. However, we show that the conventional feature matching objective with L2-distance is inherently biased toward reconstructing dominant spectral directions of the teacher representation, while under-optimizing low-variance directions that often contain task-relevant information. To address this, we propose Spectrum-Balanced Feature Matching, SpecMatch, a simple objective that adaptively emphasizes under-optimized spectral directions while preserving the relative importance of dominant directions. SpecMatch is easy to implement and introduces negligible computational overhead. Extensive experiments on image recognition demonstrate that SpecMatch consistently improves downstream adaptation across diverse tasks, including image classification, anomaly detection, medical image analysis, and domain generalization. In particular, SpecMatch outperforms conventional feature matching in 40 of 42 teacher--student and training-setting combinations, while consistently improving over the original student model in all settings. We further demonstrate that the proposed objective generalizes beyond vision, improving downstream performance across six protein understanding tasks.

Problem

• Knowledge Distillation
• Feature Matching
• Spectrum Bias
• Representation Distillation
• Visual Foundation Models

Innovation

• Knowledge Distillation
• Feature Matching
• Spectrum Bias
• Representation Learning
• Cross-modal Generalization