🤖 AI Summary
This work proposes a novel method for predicting the generalization performance of deep neural networks without requiring a validation set. By analyzing the frequency-domain characteristics of parameter trajectories during SGD optimization, the study introduces Fourier fractal dimension as a new generalization metric and establishes a theoretical framework based on the characteristic function of Lévy-driven stochastic differential equations. Building upon this insight, the authors design a trainable frequency-domain regularization mechanism together with a tailored optimizer to actively control the fractal dimension and thereby enhance generalization. Experiments on CIFAR-10, SVHN, and MNIST demonstrate that the proposed approach significantly outperforms existing generalization measures in terms of Kendall rank correlation coefficient, offering both accurate prediction and effective optimization capabilities.
📝 Abstract
Predicting the generalization performance of deep neural networks without relying on hold-out validation data is a fundamental challenge in machine learning. While Stochastic Gradient Descent (SGD) drives the optimization of these highly parameterized models, its heavy-tailed, non-Gaussian dynamics induce complex, scale-invariant trajectories in the parameter space. In this paper, we propose a novel generalization measure based on the Fourier fractal dimension of the network's weight variations. By analyzing the characteristic function of the Lévy-driven stochastic differential equations in the frequency domain, we extract a metric that robustly captures the geometric complexity of the learning process. Furthermore, we introduce a customized Fourier-based optimizer designed to actively regularize this fractal dimension during training. Extensive empirical evaluations on the CIFAR-10, SVHN, and MNIST datasets demonstrate that our proposed Fourier generalization measure exhibits a strong correlation with the actual generalization gap. Our method achieves state-of-the-art Kendall rank correlation coefficients, outperforming a wide array of existing norm-based, margin-based, and PAC-Bayesian measures. Ultimately, this work highlights the potential of frequency-domain fractal analysis as both a powerful predictor for model generalizability and a principled foundation for developing more stable optimization algorithms.