Characterizing the Discrete Geometry of ReLU Networks

2026-06-05

AI Summary

This work investigates the geometric structure of the polyhedral complex induced by fully connected ReLU networks in input space, which underpins their nonlinear behavior. Bridging discrete geometry, graph theory, and deep learning theory, the study establishes—for the first time—that the adjacency graph of this complex has an average degree bounded above by twice the input dimension and a diameter admitting a constant upper bound independent of dimensionality. Through theoretical analysis complemented by experiments on both synthetic and real-world data, the authors reveal that the ReLU-induced complex exhibits constrained local connectivity and global compactness. These findings overcome the interpretability barrier posed by the exponential growth in the number of linear regions and elucidate intrinsic geometric regularities of ReLU networks even in high-dimensional settings.

Abstract

It is well established that ReLU networks define continuous piecewise-linear functions, and that their linear regions are polyhedra in the input space. These regions form a complex that fully partitions the input space. The way these regions fit together is fundamental to the behavior of the network, as nonlinearities occur only at the boundaries where these regions connect. However, relatively little is known about the geometry of these complexes beyond bounds on the total number of regions, and calculating the complex exactly is intractable for most networks. In this work, we prove new theoretical results about these complexes that hold for all fully-connected ReLU networks, specifically about their connectivity graphs in which nodes correspond to regions and edges exist between each pair of regions connected by a face. We find that the average degree of this graph is upper bounded by twice the input dimension regardless of the width and depth of the network, and that the diameter of this graph has an upper bound that does not depend on input dimension, despite the number of regions increasing exponentially with input dimension. We corroborate our findings through experiments with networks trained on both synthetic and real-world data, which provide additional insight into the geometry of ReLU networks. Code to reproduce our results can be found at https://github.com/bl-ake/ICLR-2026.

Problem

• ReLU networks
• piecewise-linear functions
• linear regions
• discrete geometry
• connectivity graph

Innovation

• ReLU networks
• piecewise-linear functions
• linear regions
• connectivity graph
• discrete geometry