The Geometry of ReLU Networks through the ReLU Transition Graph

2025-05-16

AI Summary

Limited understanding of the structural properties of ReLU neural networks hinders theoretical analysis of their expressivity, generalization, and robustness. Method: We propose the ReLU Transition Graph (RTG)—a combinatorial graph model where nodes represent linear regions of the network and directed edges correspond to single-neuron activation flips—unifying the piecewise-linear geometric nature of ReLU networks. Contribution/Results: We establish the first theoretical connections between RTG topology and key learning-theoretic properties: deriving tight upper bounds on RTG size and diameter; proving that the RTG of any ReLU network is necessarily connected; and introducing RTG entropy and average degree as interpretable, computationally tractable generalization error criteria. Leveraging tools from combinatorial graph theory, computational geometry, and VC-dimension analysis—and empirically validating across diverse architectures and data distributions—we find a significant negative correlation between RTG entropy and test error. This provides a novel, principled foundation for network compression and geometric regularization.

Technology Category

Machine Learning: Probabilistic Circuits and Graphical ModelsReasoning under Uncertainty: Graphical ModelsKnowledge Representation and Reasoning: Computational Complexity of Reasoning

Application Category

Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphsWeb Mining and Content Analysis: Models for Web evolutionSocial Networks and Social Media: Influence propagation, information diffusion, and the prediction on networks

Abstract

We develop a novel theoretical framework for analyzing ReLU neural networks through the lens of a combinatorial object we term the ReLU Transition Graph (RTG). In this graph, each node corresponds to a linear region induced by the network's activation patterns, and edges connect regions that differ by a single neuron flip. Building on this structure, we derive a suite of new theoretical results connecting RTG geometry to expressivity, generalization, and robustness. Our contributions include tight combinatorial bounds on RTG size and diameter, a proof of RTG connectivity, and graph-theoretic interpretations of VC-dimension. We also relate entropy and average degree of the RTG to generalization error. Each theoretical result is rigorously validated via carefully controlled experiments across varied network depths, widths, and data regimes. This work provides the first unified treatment of ReLU network structure via graph theory and opens new avenues for compression, regularization, and complexity control rooted in RTG analysis.

Problem

• Analyzing ReLU networks via the ReLU Transition Graph (RTG) structure
• Connecting RTG geometry to expressivity, generalization, and robustness
• Providing graph-theoretic interpretations for VC-dimension and network properties

Innovation

• Introduces ReLU Transition Graph for network analysis
• Connects RTG geometry to network performance metrics
• Validates theory with controlled experiments rigorously