Discrete Functional Geometry of ReLU Networks via ReLU Transition Graphs

2025-09-03

AI Summary

The generalization mechanism of deep ReLU networks remains poorly understood. Method: We propose the ReLU Transition Graph (RTG), a novel graph-theoretic model where nodes represent linear activation regions and edges encode single-neuron activation flips, thereby capturing the discrete geometric structure of the network function. Contributions/Results: We introduce— for the first time in ReLU network analysis—region entropy, spectral gap of the RTG, and edge KL divergence. Leveraging random matrix theory and spectral graph theory, we derive new theoretical bounds on model capacity and generalization. We establish that: (i) region entropy saturates under over-parameterization; (ii) the RTG’s spectral gap is positively correlated with generalization performance; and (iii) edge KL divergence quantifies functional sensitivity. Empirical results confirm that RTG connectivity and smoothness improve during training, validating its efficacy as a geometric proxy for generalization ability.

Technology Category

Machine Learning: Deep Learning TheoryReasoning under Uncertainty: Graphical ModelsKnowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphsWeb Mining and Content Analysis: Models for Web evolutionSocial Networks and Social Media: Social media analysis through the lenses of networks

Abstract

We extend the ReLU Transition Graph (RTG) framework into a comprehensive graph-theoretic model for understanding deep ReLU networks. In this model, each node represents a linear activation region, and edges connect regions that differ by a single ReLU activation flip, forming a discrete geometric structure over the network's functional behavior. We prove that RTGs at random initialization exhibit strong expansion, binomial degree distributions, and spectral properties that tightly govern generalization. These structural insights enable new bounds on capacity via region entropy and on generalization via spectral gap and edge-wise KL divergence. Empirically, we construct RTGs for small networks, measure their smoothness and connectivity properties, and validate theoretical predictions. Our results show that region entropy saturates under overparameterization, spectral gap correlates with generalization, and KL divergence across adjacent regions reflects functional smoothness. This work provides a unified framework for analyzing ReLU networks through the lens of discrete functional geometry, offering new tools to understand, diagnose, and improve generalization.

Problem

• Modeling deep ReLU networks' discrete geometric structure via activation regions
• Proving expansion and spectral properties govern network generalization behavior
• Analyzing capacity and generalization through region entropy and spectral gaps

Innovation

• Graph-theoretic model for deep ReLU networks
• Nodes represent linear activation regions
• Edges connect regions with single ReLU flip