🤖 AI Summary
This work addresses the rapid information decay in deep L-Lipschitz networks caused by vanishing output variance at initialization. The authors propose a novel weight parameterization based on LDLT decomposition and, for the first time, combine zonal polynomial expectations with James’ theorem to derive a closed-form expression for the output variance under Gaussian initialization. By leveraging properties of the Wishart distribution, moment approximations, and combinatorial expansions, they optimize the initialization scale. Theoretical analysis shows that the proposed parameterization, coupled with a $10/\sqrt{n}$ scaling factor, significantly increases output variance from 0.41 to 0.9. Although He initialization still yields better empirical performance on the Higgs dataset, this study establishes the first rigorous theoretical framework for variance control in L-Lipschitz architectures and provides a tailored initialization strategy.
📝 Abstract
We analyze initialization dynamics for LDLT-based $\mathcal{L}$-Lipschitz layers by deriving the exact marginal output variance when the underlying parameter matrix $W_0\in \mathbb{R}^{m\times n}$ is initialized with IID Gaussian entries $\mathcal{N}(0,\sigma^2)$. The Wishart distribution, $S=W_0W_0^\top\sim\mathcal{W}_m(n,\sigma^2 \boldsymbol{I}_m)$, used for computing the output marginal variance is derived in closed form using expectations of zonal polynomials via James'theorem and a Laplace-integral expansion of $(\alpha \boldsymbol{I}_m+S)^{-1}$. We develop an Isserlis/Wick-based combinatorial expansion for $\operatorname{\mathbb{E}}\left[\operatorname{tr}(S^k)\right]$ and provide explicit truncated moments up to $k=10$, which yield accurate series approximations for small-to-moderate $\sigma^2$. Monte Carlo experiments confirm the theoretical estimates. Furthermore, empirical analysis was performed to quantify that, using current He or Kaiming initialization with scaling $1/\sqrt{n}$, the output variance is $0.41$, whereas the new parameterization with $10/ \sqrt{n}$ for $\alpha=1$ results in an output variance of $0.9$. The findings clarify why deep $\mathcal{L}$-Lipschitz networks suffer rapid information loss at initialization and offer practical prescriptions for choosing initialization hyperparameters to mitigate this effect. However, using the Higgs boson classification dataset, a hyperparameter sweep over optimizers, initialization scale, and depth was conducted to validate the results on real-world data, showing that although the derivation ensures variance preservation, empirical results indicate He initialization still performs better.