🤖 AI Summary
This work investigates the intrinsic mechanism enabling deep models to generalize well under label noise. Theoretically, we show that label noise primarily perturbs low-order singular components of weight matrices, while the dominant subspace governing generalization—the principal subspace spanned by top singular vectors—remains stably aligned. This yields the first rigorous subspace-level characterization of generalization robustness under label corruption. Building on this insight, we propose LIP (Low-rank Invariant Projection), a lightweight plug-in module that explicitly regularizes the principal subspace structure of weight matrices during training. Extensive experiments across diverse synthetic and real-world noisy-label benchmarks demonstrate that LIP consistently improves classification accuracy of mainstream architectures—including ResNet and ViT—without architectural modification. Our results empirically and theoretically establish principal subspace stability as the fundamental geometric principle underlying label-noise robustness.
📝 Abstract
Learning from inaccurate annotations has gained significant attention due to the high cost of precise labeling. However, despite the presence of erroneous labels, models trained on noisy data often retain the ability to make accurate predictions. This intriguing phenomenon raises a fundamental yet largely unexplored question: why models can still extract correct label information from inaccurate annotations remains unexplored. In this paper, we conduct a comprehensive investigation into this issue. By analyzing weight matrices from both empirical and theoretical perspectives, we find that label inaccuracy primarily accumulates noise in lower singular components and subtly perturbs the principal subspace. Within a certain range, the principal subspaces of weights trained on inaccurate labels remain largely aligned with those learned from clean labels, preserving essential task-relevant information. We formally prove that the angles of principal subspaces exhibit minimal deviation under moderate label inaccuracy, explaining why models can still generalize effectively. Building on these insights, we propose LIP, a lightweight plug-in designed to help classifiers retain principal subspace information while mitigating noise induced by label inaccuracy. Extensive experiments on tasks with various inaccuracy conditions demonstrate that LIP consistently enhances the performance of existing algorithms. We hope our findings can offer valuable theoretical and practical insights to understand of model robustness under inaccurate supervision.