A Statistical Difference between Single-Layer Learning and Hierarchical Learning in Wide Neural Networks

2026-07-25

AI Summary

This work investigates the fundamental differences between single-layer learning—where only the output layer is trained—and layered learning—involving joint training of input-to-hidden weights—in wide neural networks, with respect to generalization performance and the geometry of parameter space. Leveraging theoretical analysis in the infinite-width limit and integrating perspectives from large-width networks, statistical learning theory, and kernel methods, the study demonstrates that layered learning effectively eliminates singularities in parameter space and substantially reduces generalization error. In contrast to single-layer learning with fixed input-to-hidden weights, optimizing these weights not only enhances generalization but also reveals a critical mechanism: kernel optimization induced by moving away from initialization plays a pivotal role in performance improvement.

Abstract

Hierarchical neural networks are widely used in artificial intelligence, yet their mathematical properties remain incompletely understood. In the infinite-width limit, two different theoretical frameworks have been proposed. One reduces deep learning to kernel regression with a fixed kernel by assuming that the parameters remain close to their initialization, whereas the other allows the parameters to move away from their initialization, requiring the kernel itself to be optimized. In this paper, we study a three-layer neural network with a finite but large number of hidden units. We show that training the input-to-hidden weights yields a smaller generalization error than keeping them fixed. Furthermore, the latter setting exhibits singularities in the parameter space, whereas the former does not. These findings indicate that singularities play an essential role even in wide neural networks.

Problem

• hierarchical learning
• single-layer learning
• wide neural networks
• generalization error
• singularities

Innovation

• hierarchical learning
• wide neural networks
• generalization error
• parameter singularities
• kernel optimization