Optimal Rates for Generalization of Gradient Descent Methods with Deep Neural Networks

2026-06-04

AI Summary

This work addresses a critical gap in the theoretical understanding of generalization in deep ReLU networks by establishing minimax-optimal excess risk bounds for overparameterized architectures trained via gradient descent (GD) and stochastic gradient descent (SGD). Building upon the neural tangent kernel (NTK) framework and incorporating structural assumptions on deep ReLU networks, the authors combine tools from statistical learning theory and optimization analysis to derive, for the first time, generalization guarantees that match the minimax optimal rate. Under conditions where network width grows polynomially with both depth and sample size, they prove that GD and SGD achieve generalization rates comparable to those of optimal kernel methods, thereby providing a rigorous theoretical foundation for the empirical success of deep learning and filling a key void in existing generalization theory.

Abstract

Recent progress has been made in understanding the statistical generalization performance of gradient descent methods for overparameterized neural networks within the neural tangent kernel (NTK) regime. However, most of the existing work on regression problems is limited to shallow network architectures, leaving a notable gap in the theory of deep neural networks. This paper addresses this gap by presenting a comprehensive generalization analysis for deep ReLU networks trained using gradient descent (GD) and stochastic gradient descent (SGD). Specifically, we establish the first known minimax-optimal rates of excess population risk for both GD and SGD with deep ReLU networks, under the assumption that the network width scales polynomially with respect to the network depth and training sample size. Our results demonstrate that with sufficient width, gradient descent methods for deep ReLU networks can achieve optimal generalization rates on par with kernel methods.

Problem

• generalization
• deep neural networks
• gradient descent
• ReLU networks
• excess population risk

Innovation

• deep ReLU networks
• gradient descent
• generalization rates
• neural tangent kernel
• minimax optimality