🤖 AI Summary
This study investigates the generalization of over-parameterized neural networks that defy classical statistical learning theory by achieving strong performance despite having far more parameters than training samples. Through systematic PyTorch experiments on CIFAR-10 and MNIST with multiple random seeds, the work examines how optimization dynamics and implicit regularization—mediated by minibatch SGD behavior, loss landscape geometry, the neural tangent kernel (NTK) mechanism, double descent phenomena, and the lottery ticket hypothesis—affect generalization. The findings reveal that minibatch training converges to flatter minima, evidenced by an 11.8-fold reduction in the top Hessian eigenvalues, yielding a 1.61% improvement in test accuracy. Moreover, sparse subnetworks retaining only 10% of the original parameters incur less than 1.15% performance loss upon retraining. These results underscore the critical interplay among architecture, optimizer, and loss landscape geometry in enabling generalization in high-dimensional models.
📝 Abstract
Classical statistical learning theory predicts that overparameterized models should exhibit severe overfitting, yet modern deep neural networks with far more parameters than training samples consistently generalize well. This contradiction has become a central theoretical question in machine learning.
This study investigates the role of optimization dynamics and implicit regularization in enabling generalization in overparameterized neural networks through controlled experiments. We examine stochastic gradient descent (SGD) across batch sizes, the geometry of flat versus sharp minima via Hessian eigenvalue estimation and weight perturbation analysis, the Neural Tangent Kernel (NTK) regime through wide-network experiments, double descent across model scales, and the Lottery Ticket Hypothesis through iterative magnitude pruning. All experiments use PyTorch on CIFAR-10 and MNIST with multiple random seeds.
Our findings demonstrate that generalization is strongly influenced by the interaction between network architecture, optimization algorithms, and loss landscape geometry. Smaller batch sizes consistently produced lower test error and flatter minima, with an 11.8x difference in top Hessian eigenvalue between small-batch and large-batch solutions corresponding to 1.61 percentage points higher test accuracy. Sparse subnetworks retaining only 10% of parameters achieved within 1.15 percentage points of full model performance when retrained from their original initialization. These results highlight the need for revised learning-theoretic frameworks capable of explaining generalization in high-dimensional model regimes.