Global Exponential Convergence of Two-Layer Linear Network Training

2026-10-06

AI Summary

This study addresses the lack of explicit global exponential convergence rates and finite-width stability guarantees for wide two-layer linear networks trained under pairwise learning (PL) loss. By establishing covariance dynamics grounded in the Bures flow and mean-field conservation laws, we derive an explicit convergence rate conditioned on the initial spectral gap. The core contribution is the first derivation of an explicit linear convergence rate in the rich scaling regime, which we further extend to deep ResNets and Heavy-ball momentum settings while proving finite-width stability. Numerical experiments validate the theoretical predictions, precisely delineating step-size intervals and damping windows, thereby confirming the efficacy of the proposed covariance geometry analysis.

Abstract

We prove global exponential (linear) convergence with an explicit rate in the rich scaling for wide two-layer linear networks trained with smooth Polyak-Lojasiewicz predictor losses. Gradient flow in the factors closes exactly in terms of a finite-dimensional Bures flow of the neuron law covariance, in which the predictor dynamics are preconditioned by hidden covariance blocks. Mean-field conservation laws provide uniform spectral lower bounds on the hidden preconditioning blocks when the initial covariance satisfies a spectral support gap condition. This condition encompasses positive definiteness while still allowing for singular initializations. For an initial covariance $Σ_0 = σ^2 \mathrm{Id}$, the loss converges to the global minimum with linear rate at least $4σ^2κ$, where $κ$ is the PL constant. We establish stability of this rate under finite-width sampling, as well as global convergence of factor gradient descent for an explicit stepsize interval depending on smoothness, the initial loss, and conserved spectral margins. Our argument extends layerwise to deep linear ResNets, subject to a residual-path bound. In the case of heavy-ball momentum, training dynamics close instead over positions and velocities in terms of a lifted phase covariance. Linear convergence holds under an explicit condition on the energy and damping, specifying a window of admissible dampings. For two-scale white initializations, this interval is nonempty for sufficiently large position scales, with a fixed initial loss gap and velocity covariance. Numerical experiments illustrate the covariance geometry and compare the predicted and observed rates.

Problem

• global exponential convergence
• two-layer linear networks
• Polyak-Lojasiewicz loss
• gradient flow
• heavy-ball momentum

Innovation

• Global Exponential Convergence
• Bures Flow
• Polyak-Lojasiewicz Loss
• Mean-field Conservation Laws
• Heavy-ball Momentum