Phase transitions reveal hierarchical structure in deep neural networks

2025-12-05

AI Summary

Apparent phenomena in deep neural network training—such as phase transitions, ubiquitous saddle points, and model connectivity—are often studied in isolation, yet they fundamentally arise from the intrinsic geometric structure of loss and error landscapes. Method: We propose a unified geometric framework that analytically links phase transitions to saddle-point distributions; design an L₂-regularized error landscape probing algorithm to uncover the geometric mechanism underlying inter-class transfer; and combine theoretical analysis with MNIST experiments, employing path optimization to connect global minima. Contribution/Results: Our approach empirically validates mode connectivity and reveals a hierarchical precision basin structure stratified by digit classes. The framework provides a novel geometric perspective on deep learning optimization dynamics and introduces computationally tractable tools for landscape analysis—advancing both theoretical understanding and practical optimization strategies.

Technology Category

• Machine Learning: Learning with Manifolds
• Search and Optimization: Learning to Search
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• Social Networks and Social Media: Social media analysis through the lenses of networks

Abstract

Training Deep Neural Networks relies on the model converging on a high-dimensional, non-convex loss landscape toward a good minimum. Yet, much of the phenomenology of training remains ill understood. We focus on three seemingly disparate observations: the occurrence of phase transitions reminiscent of statistical physics, the ubiquity of saddle points, and phenomenon of mode connectivity relevant for model merging. We unify these within a single explanatory framework, the geometry of the loss and error landscapes. We analytically show that phase transitions in DNN learning are governed by saddle points in the loss landscape. Building on this insight, we introduce a simple, fast, and easy to implement algorithm that uses the L2 regularizer as a tool to probe the geometry of error landscapes. We apply it to confirm mode connectivity in DNNs trained on the MNIST dataset by efficiently finding paths that connect global minima. We then show numerically that saddle points induce transitions between models that encode distinct digit classes. Our work establishes the geometric origin of key training phenomena in DNNs and reveals a hierarchy of accuracy basins analogous to phases in statistical physics.

Problem

• Unifies phase transitions, saddle points, and mode connectivity in DNN training
• Shows saddle points govern phase transitions in neural network learning
• Reveals hierarchical accuracy basins analogous to phases in physics

Innovation

• Phase transitions governed by saddle points in loss landscape
• L2 regularizer probes error landscape geometry for connectivity
• Saddle points induce transitions between distinct class models