Dimension-Free Rank Lifting from Random Hyperplane Arrangements

2026-09-30

AI Summary

This study addresses the problem of determining the minimum width required for a random hidden layer in neural networks to achieve rank elevation. For positively homogeneous non-polynomial activation functions, the theoretical analysis leverages techniques including Gaussian directional coupling, local hyperplane intersection, matrix concentration inequalities, and Taylor tail truncation. The primary contribution is the proof that dimension-independent bounds on the number of neurons suffice to attain full row rank, alongside the quantification of high-probability lower bounds for stable rank elevation. These results yield an exponential improvement over prior guarantees for general dimensions, provide a unified extension of stable rank theory across diverse activation functions, and establish tight bounds for rank elevation.

Abstract

We study the width required for a randomly initialized hidden layer of a neural network to achieve rank lifting. Namely, given a dataset $X \in \mathbb{R}^{m \times d}$ of $m$, $d$-dimensional input vectors separated by an angle of at least $θ$, we consider the random feature matrix $σ(XR)$, where $R$ is standard Gaussian. For positively homogeneous nonpolynomial activations, which include sign, Heaviside, ReLU, and ReLU powers among others, we prove that $$n \gtrsim \frac{1}θ\max\left\{m,\log\left(\frac{1}δ\right)\right\}$$ neurons suffice for $σ(XR)$ to have full row rank $m$ with probability at least $1-δ$. This dimension-free bound exponentially improves the previous general-dimensional guarantee for sign features (Drago et al., 2026) and is essentially tight. The proof shows that one random feature column escapes every proper subspace of $\mathbb{R}^m$ with probability $Ω(θ)$, using a coupling of nearby Gaussian directions and a local crossing of the induced hyperplane arrangement. We also study stable rank lifting, where the goal is to establish a quantitative analogue of exact rank lifting, i.e., a lower bound on the smallest eigenvalue of the empirical feature Gram matrix in high-probability. Our analysis unifies and generalizes stable rank guarantees for all $q$-homogeneous non-polynomial activations following prior work in Panigrahi et al. (2020) and Song (2026). In particular, we combine a diagonally dominant Taylor tail of the population kernel with truncation and matrix concentration, to show that for positively homogeneous nonpolynomial activations, stable rank lifting is achieved at width $$n \gtrsim C^q \frac{m}{θ^{2q+1}} \log^{2q+\frac{1}{2}}\left(\frac{m}θ\right) \log\left(\frac{m}δ\right),$$ where $q$ is the degree of the activation and $C > 0$ is some universal constant.

Problem

• rank lifting
• random features
• neural network width
• stable rank
• hyperplane arrangements

Innovation

• Rank Lifting
• Dimension-Free Bound
• Random Hyperplane Arrangements
• Stable Rank
• Homogeneous Activations