Gradient Flow Through Diagram Expansions: Learning Regimes and Explicit Solutions

2026-02-04

AI Summary

This work investigates the evolution mechanisms of gradient flow in large-scale learning and their dependence on model architecture. By constructing a formal power series expansion of the loss function and introducing Feynman-diagram-like structures to encode high-order coefficients, we establish a graph-based analytical framework for gradient flow. In the large-scale limit, combining first-order partial differential equations with the method of characteristics, we systematically reveal— for the first time—how parameter scaling, tensor order, and symmetry govern the emergence of lazy versus rich learning regimes. Focusing on higher-order tensor CP decomposition models, we derive explicit analytical solutions that identify multiple extreme learning mechanisms, including free evolution, neural tangent kernel (NTK), and under/over-parameterized mean-field dynamics. Theoretical predictions show excellent agreement with empirical experiments.

Technology Category

• Machine Learning: Deep Neural Architectures and Foundation Models
• Natural Language Processing: Learning & Optimization for NLP
• Computer Vision: Learning & Optimization for CV

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• User Modeling, Personalization and Recommendation: Practical large-scale studies of user experience
• Web Mining and Content Analysis: Models for Web evolution

Abstract

We develop a general mathematical framework to analyze scaling regimes and derive explicit analytic solutions for gradient flow (GF) in large learning problems. Our key innovation is a formal power series expansion of the loss evolution, with coefficients encoded by diagrams akin to Feynman diagrams. We show that this expansion has a well-defined large-size limit that can be used to reveal different learning phases and, in some cases, to obtain explicit solutions of the nonlinear GF. We focus on learning Canonical Polyadic (CP) decompositions of high-order tensors, and show that this model has several distinct extreme lazy and rich GF regimes such as free evolution, NTK and under- and over-parameterized mean-field. We show that these regimes depend on the parameter scaling, tensor order, and symmetry of the model in a specific and subtle way. Moreover, we propose a general approach to summing the formal loss expansion by reducing it to a PDE; in a wide range of scenarios, it turns out to be 1st order and solvable by the method of characteristics. We observe a very good agreement of our theoretical predictions with experiment.

Problem

• gradient flow
• tensor decomposition
• learning regimes
• scaling limits
• nonlinear dynamics

Innovation

• gradient flow
• diagram expansion
• Feynman diagrams
• learning regimes
• tensor decomposition