Stable initialization without the CLT

2026-09-24

AI Summary

This study addresses the inherent approximation errors and inter-layer coupling arising from the reliance of deep network initialization on Central Limit Theorem (CLT)-based distribution approximations. Focusing on sinusoidal activation functions, this work is the first to exploit their periodic symmetry to eliminate conventional distributional approximations. It proposes a uniform phase initialization algorithm that, when integrated with μP width-scaling theory, achieves fully layer-decoupled and CLT-free stable initialization. Empirical evaluations demonstrate that the proposed method surpasses existing state-of-the-art performance on image and audio fitting tasks. Notably, it matches the strongest baselines without requiring hyperparameter tuning while natively supporting width scaling.

Abstract

Successful training of deep neural networks is highly dependent on the distribution of the initial weights. If the weights are too large, network training blows up; if they are too small, the model fails to learn features. Stable initialization is the optimal moderation between these two extremes. The conventional theory of random networks uses the Central Limit Theorem to control inter-neuron dependencies, which introduces distributional approximation error and coupling between layers. For networks with sine activations, we derive the uniform-phase initialization, which obviates distributional approximation and fully decouples the layers. Ours is the first work to use the sine function's periodic symmetry. Models trained with the uniform-phase initialization outperform the state of the art in neural representation tasks like image and audio fitting. We find that our untuned models are competitive with the best-tuned baselines from previous work and support $μ$P width scaling.

Problem

• stable initialization
• deep neural networks
• Central Limit Theorem
• distributional approximation error
• layer coupling

Innovation

• Uniform-phase initialization
• Sine activation function
• Central Limit Theorem
• Layer decoupling
• Neural representation