Gradient Flow Equations for Deep Linear Neural Networks: A Survey from a Network Perspective

2025-11-13

AI Summary

This work investigates the gradient flow dynamics and loss landscape geometry of deep linear neural networks under quadratic loss. Methodologically, it models gradient flow as an isospectral matrix differential equation and integrates Lyapunov stability analysis, algebraic invariant theory, and quotient space modeling. Theoretically, it rigorously proves that the loss function has no spurious local minima—its critical points are exclusively global minima and saddle points. It introduces a unique representation of critical values in the quotient space, precisely characterizing gradient flow invariant sets and stable submanifolds. Furthermore, it首次 (for the first time) fully uncovers the hierarchical geometric structure of level sets, establishes a complete classification of critical points, and elucidates the convergence mechanism: gradient flow converges almost surely to a global minimum, with learning corresponding to sequential alignment and capture of singular values of the weight matrices.

Technology Category

• Machine Learning: Learning with Manifolds
• Natural Language Processing: Learning & Optimization for NLP
• Search and Optimization: Learning to Search

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Web Mining and Content Analysis: Machine learning and data science for the Web
• Social Networks and Social Media: Social media analysis through the lenses of networks

Abstract

The paper surveys recent progresses in understanding the dynamics and loss landscape of the gradient flow equations associated to deep linear neural networks, i.e., the gradient descent training dynamics (in the limit when the step size goes to 0) of deep neural networks missing the activation functions and subject to quadratic loss functions. When formulated in terms of the adjacency matrix of the neural network, as we do in the paper, these gradient flow equations form a class of converging matrix ODEs which is nilpotent, polynomial, isospectral, and with conservation laws. The loss landscape is described in detail. It is characterized by infinitely many global minima and saddle points, both strict and nonstrict, but lacks local minima and maxima. The loss function itself is a positive semidefinite Lyapunov function for the gradient flow, and its level sets are unbounded invariant sets of critical points, with critical values that correspond to the amount of singular values of the input-output data learnt by the gradient along a certain trajectory. The adjacency matrix representation we use in the paper allows to highlight the existence of a quotient space structure in which each critical value of the loss function is represented only once, while all other critical points with the same critical value belong to the fiber associated to the quotient space. It also allows to easily determine stable and unstable submanifolds at the saddle points, even when the Hessian fails to obtain them.

Problem

• Analyzing gradient flow dynamics in deep linear neural networks without activation functions
• Characterizing loss landscapes with global minima and saddle points but no local minima
• Investigating critical point structures using adjacency matrix representations and quotient spaces

Innovation

• Analyzes gradient flow in linear networks
• Uses adjacency matrix representation for ODEs
• Reveals quotient space structure of loss landscape