🤖 AI Summary
This study investigates whether a single-hidden-layer multilayer perceptron (MLP) of fixed width can, through parameter reconfiguration, implicitly absorb its skip connections and thereby eliminate the explicit residual structure. By combining homogeneity analysis, linearization arguments, matrix algebra, and perspectives from function spaces and probability measures, the work establishes necessary and sufficient conditions under which such absorption is possible. It proves that for activation functions like ReLU and GELU, absorption holds only under specific algebraic constraints, whereas for ReLU², SwiGLU, and GeGLU, it is generally infeasible under generic weight configurations. These findings imply that, at equal width, MLPs with and without skip connections typically represent function classes that are almost everywhere disjoint, and single-block absorption is achievable only under non-generic conditions.
📝 Abstract
We study when a skip connection around a single-hidden-layer MLP can be absorbed into a residual-free MLP of the same width. We first show that for any architecture whose skip branch is an invertible linear map (including Hyper-Connections and their manifold-constrained variants), the problem reduces to the identity skip case. For homogeneous activations of degree $k \neq 1$, such as ReLU$^2$ and ReGLU, absorption is unconditionally impossible by a degree argument. For gated activations whose gate is differentiable at the origin with $g(0) = 0$, including SwiGLU and GeGLU, a linearization argument gives the same conclusion. These impossibility results extend to arbitrary depth: a composition of $L$ residual blocks using such activations cannot be replicated by any composition of $L$ residual-free blocks of the same width. For ungated ReLU and GELU, the situation is richer. For generic weight matrices, absorption holds at the single-block level if and only if there exists an index set $S$ of size at least $d$ such that $W_{\mathrm{down}}[:,S]\,W_{\mathrm{up}}[S,:] = -I_d$. This condition is non-generic (it fails with probability one under continuous weight distributions), so skip-connected and residual-free MLPs of the same width represent generically disjoint function classes. Whether this disjointness persists for deep compositions of ReLU or GELU blocks remains open.