Shallower ReLU Network Representations via Exact Linear Algebra

2026-07-22

AI Summary

This work investigates the minimal-depth ReLU neural network architectures capable of exactly representing the max function and general continuous piecewise linear (CPWL) functions. By reformulating the problem as a linear algebraic one over the rational field and leveraging symmetry reduction together with finite linear system solving, the authors construct networks with improved depth bounds. Their main contributions include the first proof that the max function over $ n \leq 10 $ inputs can be exactly realized by a ReLU network with only two hidden layers; for $ n > 10 $, they establish a depth upper bound of $ \lceil \log_5(n/2) \rceil + 1 $, which improves upon the previously known $ \log_3 $ bound. Furthermore, they extend these results to show that any CPWL function in dimension $ d \leq 9 $ can also be exactly represented by a two-hidden-layer ReLU network.

Abstract

We prove that the maximum of $n$ real numbers is exactly representable by a ReLU network with two hidden layers for every $n\le 10$. The constructions are obtained by reducing the problem to exact rational linear algebra: after a symmetry reduction, the necessary cancellations are encoded in finite linear systems over $\mathbb{Q}$, which we solve and verify computationally. The representation of $\max_{10}$ has a structured first hidden layer consisting only of pairwise maxima, a feature that allows it to be recursively substituted into larger networks. We use this to show that for every $n>10$, the maximum $\max_{n}$ can be exactly represented with $\lceil{\log_5 (n / 2)\rceil}+1 < \log_5(n) +1.5694$ hidden layers. Via the generalized hinging-hyperplane representation [Wang, Sun, IEEE Trans. Inf. Theory 2005], the same depth bound holds for all continuous piecewise-linear functions on $\mathbb{R}^d$, with $d+1$ in place of $n$. In particular, every continuous piecewise-linear function on $\mathbb{R}^d$ for $d\le 9$ admits a two-hidden-layer ReLU representation. Our results improve on [Bakaev, Brunck, Hertrich, Stade, Yehudayoff, STOC'26]. In that work, the authors established a two-hidden-layer representation for $\max_{5}$ and an upper bound of $\lceil{\log_3 (n-2)\rceil}+1$ hidden layers for $\max_{n}$.

Problem

• ReLU networks
• max function
• piecewise-linear functions
• network depth
• exact representation

Innovation

• ReLU networks
• exact representation
• piecewise-linear functions
• linear algebra
• network depth