🤖 AI Summary
This work proposes a novel data quality diagnostic method for detecting label noise in training datasets by analyzing the tail index (α) of the eigenvalue distribution of weight matrices in neural network bottleneck layers. The study establishes, for the first time, a strong correlation between the tail index and the level of label noise, positioning α as a dedicated metric for data quality assessment rather than generalization analysis. Leveraging random matrix theory, spectral analysis, and tail index estimation, the approach achieves an R² of 0.984 in predicting test accuracy across 21 noise levels—significantly outperforming conventional methods. Furthermore, on the CIFAR-10N benchmark, it successfully identifies 9% of artificially introduced annotation errors with only a 3% false positive rate.
📝 Abstract
We investigate whether spectral properties of neural network weight matrices can predict test accuracy. Under controlled label noise variation, the tail index alpha of the eigenvalue distribution at the network's bottleneck layer predicts test accuracy with leave-one-out R^2 = 0.984 (21 noise levels, 3 seeds per level), far exceeding all baselines: the best conventional metric (Frobenius norm of the optimal layer) achieves LOO R^2 = 0.149. This relationship holds across three architectures (MLP, CNN, ResNet-18) and two datasets (MNIST, CIFAR-10). However, under hyperparameter variation at fixed data quality (180 configurations varying width, depth, learning rate, and weight decay), all spectral and conventional measures are weak predictors (R^2 < 0.25), with simple baselines (global L_2 norm, LOO R^2 = 0.219) slightly outperforming spectral measures (tail alpha, LOO R^2 = 0.167). We therefore frame the tail index as a data quality diagnostic: a powerful detector of label corruption and training set degradation, rather than a universal generalization predictor. A noise detector calibrated on synthetic noise successfully identifies real human annotation errors in CIFAR-10N (9% noise detected with 3% error). We identify the information-processing bottleneck layer as the locus of this signature and connect the observations to the BBP phase transition in spiked random matrix models. We also report a negative result: the level spacing ratio <r> is uninformative for weight matrices due to Wishart universality.