parameter initialization

Function-parameterized neural networks are highly sensitive to initialization, and conventional data-agnostic initialization schemes often fail to capture the structural characteristics of target signals, leading to slow convergence and unstable performance. This work proposes a prior-guided initialization strategy that, for the first time, integrates data-driven spectral priors into both network initialization and architecture design. Specifically, fast Fourier transform (FFT) is employed to extract seasonal priors that inform model depth and initial state, while residual regression is used to parameterize trend components. Without altering the training procedure, the proposed method significantly accelerates convergence, reduces performance variance, and improves computational efficiency across both synthetic and real-world datasets. Notably, it maintains reconstruction accuracy even when using a lower-dimensional encoder, consistently outperforming standard initialization approaches.

To address the lack of structure-agnostic, theoretically interpretable, and robustly generalizable initialization methods for neural networks, this paper proposes a layer-wise initialization strategy grounded in shrinkage estimation, coupled with a smoothing learning mechanism. It is the first work to introduce shrinkage estimation into weight initialization—requiring no prior assumptions about parameter distributions and independent of network topology—while providing rigorous theoretical guarantees. The smoothing learning component effectively mitigates gradient discontinuities and optimization oscillations. Empirical evaluation across multiple synthetic datasets demonstrates that, compared to Xavier and He initialization, the proposed method reduces early-training loss fluctuation by 47%, improves training stability by 32%, accelerates convergence, and significantly enhances generalization performance. The core contribution is the establishment of the first structure-agnostic, theoretically interpretable, and hyperparameter-free universal initialization framework.

Existing initialization methods for residual networks—such as Fixup—struggle to enforce strict identity mappings simultaneously across both the main and shortcut branches, weakening inductive bias and compromising training stability. To address this, we propose IDInit, a fully identity-based initialization scheme. Its core innovation lies in employing padded quasi-identity matrices to overcome rank constraints inherent in non-square weight tensors, thereby enabling dual identity initialization of both the main-path layers and the shortcut branch within each residual block for the first time. We provide theoretical analysis establishing convergence guarantees under SGD, and further enhance robustness via high-order tensor extensions and dynamic dead-neuron compensation. Extensive experiments on large-scale datasets and deep architectures demonstrate that IDInit significantly improves training stability and convergence speed, while achieving superior generalization performance compared to state-of-the-art baselines including Fixup and ReZero.

Pre-trained models struggle to adapt efficiently to diverse deployment sizes (e.g., varying depth or width), necessitating size-specific initialization strategies. Method: This paper proposes a multi-task-inspired framework for variable-size model initialization. Its core innovation is a size-agnostic shared weight template—constructed via Learngene-based knowledge distillation with Kronecker-structured constraints—and lightweight learnable scalers that enable consistent cross-size initialization. The template-scaler co-design supports zero-shot transfer to diverse downstream tasks, while scalers require only minimal data for adaptation. Contribution/Results: Experiments demonstrate state-of-the-art initialization performance across multiple depth- and width-varied architectures. The method significantly enhances few-shot adaptability and cross-task generalization, offering a scalable, data-efficient solution for deploying models of heterogeneous sizes without retraining from scratch.

In large-scale pretraining, learning rate scheduling critically influences both training efficiency and model performance. This work proposes two paradigms—Fitting and Transfer. The Fitting paradigm establishes, for the first time, a scaling law for learning rate search factors, reducing hyperparameter tuning complexity from O(n³) to O(n·C_D·C_η). The Transfer paradigm extends μTransfer to Mixture-of-Experts (MoE) architectures and generalizes it across multiple hyperparameter dimensions, including depth, weight decay, and token length. Empirical results demonstrate that while μTransfer exhibits limited scalability in large-scale settings, the Fitting paradigm—grounded in the derived scaling law—offers superior scalability and practicality, providing a systematic guideline for hyperparameter tuning in industrial-scale pretraining.

This study investigates whether default hyperparameters in machine learning libraries serve as effective initial points for Bayesian optimization to accelerate convergence. The authors conduct the first large-scale empirical evaluation by initializing optimization with samples drawn from a truncated Gaussian distribution centered around default values and comparing this strategy against uniform random initialization. Experiments span three optimization frameworks—BoTorch, Optuna, and Scikit-Optimize—combined with Random Forest, SVM, and MLP models across five standard datasets. Results show that default hyperparameters do not yield statistically significant performance improvements (p = 0.141–0.908), and any early advantage they confer dissipates as optimization progresses. These findings suggest that default values lack informative prior knowledge, challenging the common heuristic of using them as starting points in hyperparameter optimization.

This work addresses the rapid information decay in deep L-Lipschitz networks caused by vanishing output variance at initialization. The authors propose a novel weight parameterization based on LDLT decomposition and, for the first time, combine zonal polynomial expectations with James’ theorem to derive a closed-form expression for the output variance under Gaussian initialization. By leveraging properties of the Wishart distribution, moment approximations, and combinatorial expansions, they optimize the initialization scale. Theoretical analysis shows that the proposed parameterization, coupled with a $10/\sqrt{n}$ scaling factor, significantly increases output variance from 0.41 to 0.9. Although He initialization still yields better empirical performance on the Higgs dataset, this study establishes the first rigorous theoretical framework for variance control in L-Lipschitz architectures and provides a tailored initialization strategy.

This work addresses the high cost of hyperparameter tuning during model upscaling and the lack of theoretical guarantees in existing small-to-large model extrapolation methods. The authors propose a general upscaling framework grounded in μP (maximal update parametrization) theory and architectural equivalence across arbitrary widths, establishing a rigorous equivalence between a narrow base model and its widened counterpart. This equivalence enables efficient knowledge transfer and accelerated training. By extending μTransfer to support upscaling scenarios, the method provides, for the first time, a theoretically sound foundation for hyperparameter transfer from small to large models. Extensive experiments across multiple real-world datasets and mainstream architectures demonstrate the approach’s effectiveness, significantly reducing training costs and improving convergence speed for large models.

This work proposes a self-supervised pretraining-based initialization strategy to address the limitations of conventional neural network initialization methods such as Xavier and Kaiming, which rely on random sampling and neglect structural information relevant to the optimization process. These traditional approaches often lead to slow convergence and pronounced low-frequency bias in high-resolution tasks. The proposed method uniquely leverages white noise as a self-supervised signal to generate structured initial weights by fitting random noise, requiring neither additional data nor architectural modifications. This approach effectively mitigates the low-frequency bias inherent in Implicit Neural Representations (INRs) and Deep Image Priors (DIP), significantly accelerating convergence and enhancing stability in high-resolution settings while enabling earlier capture of high-frequency components for more efficient optimization.

This work addresses the high computational cost of conventional score-based diffusion models, which typically initialize the reverse process from a Gaussian distribution and require a large number of discrete sampling steps. The authors theoretically demonstrate, for the first time, that the initialization of the reverse process critically influences sampling efficiency. They propose a learnable initialization strategy that is agnostic to the underlying model architecture, network design, and discretization scheme, directly minimizing the initial Kullback–Leibler (KL) divergence error. Grounded in KL convergence analysis and dynamical modeling of Variance Exploding (VE) diffusion processes, the method achieves comparable or superior generation quality on both synthetic data and standard benchmarks using significantly fewer sampling steps.