Understanding the Subspace Stabilization of the Hessian and Gradient Covariance Matrix

2026-09-25

AI Summary

This study addresses the fundamental limitation of existing theoretical explanations for the stability of the top Hessian subspace during neural network training, which inherently rely on the magnitude of parameter changes. To overcome this, we propose a measure of subspace evolution instability that is independent of parameter change magnitude. By leveraging an inter-class and intra-class decomposition of the gradient covariance matrix, we establish an explicit approximation theory for the top Hessian subspace. Furthermore, eigenvalue separation analysis under gradient flow reveals the underlying evolutionary dynamics. Our key contribution demonstrates that the spectral gap between outlier and bulk eigenvalues induces slow approximate evolution, thereby providing a rigorous theoretical explanation for the observed stability phenomenon of the top Hessian subspace in deep learning optimization.

Abstract

The phenomenon of the top subspace stabilization of the Hessian matrix is an surprising and critical aspect in study of the second-order information of neural network training. Prior work argues that the top subspace of the Hessian stabilizes by measuring the overlap between the top subspaces of the step-wise Hessian, and explains this stabilization with diminishing parameter change in the late phase of training. In this paper, we define a new instability metric for the subspace evolution, and use it to detect subspace stabilization that is independent of the magnitude of parameter change. In the meantime, we observe that the gradient covariance matrix has a similar property of its top subspace to the Hessian. By using a between-class and within-class decomposition of the gradient covariance matrix, we identify an explicit form that gives a near-perfect approximation of the top-$(C-1)$ subspace of the Hessian and the gradient covariance matrix. In the gradient flow set-up, we show that the slow evolution of the idenfied approximation is due to the separation between the outlier and the bulk eigenvalues of the Hessian matrix, thus providing an explanation to the phenomenon of the top subspace stabilization of the Hessian matrix.

Problem

• Hessian matrix
• subspace stabilization
• gradient covariance matrix
• neural network training
• eigenvalue separation

Innovation

• Hessian matrix
• gradient covariance matrix
• subspace stabilization
• instability metric
• eigenvalue separation