Rethinking Self-Distillation: Label Averaging and Enhanced Soft Label Refinement with Partial Labels

2024-02-16

AI Summary

This work investigates linear probing with a fixed feature extractor and reveals that, when conventional feature learning interpretations fail, self-distillation fundamentally performs label averaging guided by feature correlations encoded in the Gram matrix—yielding implicit label smoothing and enhanced noise robustness. We theoretically establish that under mild conditions, iterative self-distillation achieves 100% population accuracy. To avoid the computational overhead of multiple iterations, we propose PLL, an efficient single-round self-distillation paradigm: it constructs refined partial labels using the top-two softmax outputs from a teacher model. Leveraging spectral graph theory and deriving generalization error bounds, PLL significantly outperforms multi-round self-distillation on benchmarks including CIFAR-10/100, achieving state-of-the-art noise robustness—especially under high label noise—while maintaining theoretical rigor and practical efficiency.

Technology Category

• Machine Learning: Online Learning & Bandits
• Search and Optimization: Learning to Search
• Natural Language Processing: Safety and Robustness

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs

Abstract

We investigate the mechanisms of self-distillation in multi-class classification, particularly in the context of linear probing with fixed feature extractors where traditional feature learning explanations do not apply. Our theoretical analysis reveals that multi-round self-distillation effectively performs label averaging among instances with high feature correlations, governed by the eigenvectors of the Gram matrix derived from input features. This process leads to clustered predictions and improved generalization, mitigating the impact of label noise by reducing the model's reliance on potentially corrupted labels. We establish conditions under which multi-round self-distillation achieves 100% population accuracy despite label noise. Furthermore, we introduce a novel, efficient single-round self-distillation method using refined partial labels from the teacher's top two softmax outputs, referred to as the PLL student model. This approach replicates the benefits of multi-round distillation in a single round, achieving comparable or superior performance--especially in high-noise scenarios--while significantly reducing computational cost.

Problem

• Explores self-distillation in multi-class classification
• Mitigates label noise impact via label averaging
• Introduces efficient single-round self-distillation method

Innovation

• Label averaging enhances generalization
• Partial labels refine soft label distillation
• Single-round self-distillation reduces computational cost