initialization strategies

Learning rate warmup empirically accelerates large-model training, yet its theoretical underpinnings remain poorly understood. This paper bridges this gap by introducing a novel class of generalized smoothness assumptions grounded in optimization theory, enabling the first rigorous characterization of how warmup mitigates gradient mismatch during early training stages. We prove that, under these assumptions, warmup-equipped gradient descent (GD) and stochastic gradient descent (SGD) achieve convergence rates accelerated by a factor of Θ(T) over their non-warmup counterparts—substantially improving upon classical O(1/√T) or O(1/T) bounds. Our analysis comprehensively covers both deterministic and stochastic optimization settings. Extensive experiments on canonical neural networks validate the theoretical predictions. By establishing a principled analytical framework linking warmup to gradient alignment and curvature adaptation, this work resolves the longstanding disconnect between the empirical success and theoretical opacity of warmup, offering a new paradigm for designing adaptive learning rate schedules.

This work provides a rigorous theoretical foundation for learning rate warmup—a widely adopted yet poorly understood deep learning practice. Addressing the question of why warmup accelerates convergence, we introduce a novel theoretical framework based on generalized $(L_0, L_1)$-smoothness, which for the first time characterizes the evolution of local curvature of the loss function during early training. Leveraging this, we establish formal convergence guarantees for warmup strategies and derive tight upper and lower complexity bounds—revealing that acceleration stems from adaptive avoidance of highly curved initial regions. Our analysis applies to multilayer neural networks under both mean squared error and cross-entropy losses. Extensive experiments on language and vision models confirm that warmup significantly improves convergence speed and robustness compared to fixed learning rates.

Neural networks often struggle to efficiently acquire high-level capabilities during training, limiting learning efficiency and generalization. Method: We propose a layer-adaptive weight scaling initialization method explicitly designed to foster the emergence of such capabilities. For the first time, we formalize “emergence” as a quantifiable structural nonlinearity metric and use it to guide initialization—without requiring additional optimization steps. The method is architecture-agnostic, supporting MLPs, ConvNets, and Transformers. Contribution/Results: Experiments on image classification and machine translation demonstrate significant improvements in both test accuracy and convergence speed. Notably, the method exhibits strong robustness to the presence or absence of BatchNorm, underscoring its practical versatility and theoretical novelty. The approach is theoretically grounded, computationally lightweight, and straightforward to implement.

In large-scale pretraining, learning rate scheduling critically influences both training efficiency and model performance. This work proposes two paradigms—Fitting and Transfer. The Fitting paradigm establishes, for the first time, a scaling law for learning rate search factors, reducing hyperparameter tuning complexity from O(n³) to O(n·C_D·C_η). The Transfer paradigm extends μTransfer to Mixture-of-Experts (MoE) architectures and generalizes it across multiple hyperparameter dimensions, including depth, weight decay, and token length. Empirical results demonstrate that while μTransfer exhibits limited scalability in large-scale settings, the Fitting paradigm—grounded in the derived scaling law—offers superior scalability and practicality, providing a systematic guideline for hyperparameter tuning in industrial-scale pretraining.

To address the inefficiency and poor generalizability of manual hyperparameter tuning—particularly for learning rates—this paper proposes a dynamic online meta-optimization framework that formulates learning rate adaptation as a discounted cumulative regret minimization problem over time. The method employs a gradient-based meta-update mechanism, enabling plug-and-play integration with any first-order optimizer (e.g., SGD, Adam) to achieve decoupled, real-time, adaptive step-size optimization. Key contributions include: (i) the first formalization of meta-optimization as discounted regret minimization; and (ii) a low-complexity variant that preserves theoretical rigor while ensuring computational efficiency and strong generalization. Experiments across diverse tasks demonstrate faster convergence, enhanced robustness to initialization and task heterogeneity, competitive performance against hand-tuned optimal schedulers, and significantly lower computational overhead compared to conventional hyperparameter search methods.

This study investigates the limitations of warm-starting language models in scaling training and identifies two key obstacles: an overemphasis on preserving initial performance and a neglect of the interaction between growth factors and hyperparameters. Through systematic experiments—including ablation studies, cross-architecture validation (spanning dense MLPs and language models), scaling law fits, and multi-budget training efficiency comparisons—the work demonstrates that retaining initial performance is unnecessary and that simple, architecture-agnostic growth strategies outperform complex operators. Crucially, the research reveals for the first time an efficiency ceiling for growth factors: 2× scaling most reliably accelerates convergence under low compute budgets (<20 tokens per parameter), beyond which training from scratch becomes more efficient. These findings yield a practical, predictable guideline for effective model scaling.

This work addresses the limitations of existing sampler warm-up strategies, which rely on fixed schedules and heuristic hyperparameter tuning and often fail to adapt to complex posterior geometries. The authors propose a general adaptive warm-up framework that reformulates warm-up as an evidence-based path selection problem. Guided uniformly by an estimated posterior covariance Σ_π under a fixed computational budget, the method dynamically adjusts hyperparameters in real time. Applicable to a broad class of Hamiltonian Monte Carlo (HMC) samplers, it integrates local metric adaptation with global geometric awareness, featuring path attractivity and an explicit rejection mechanism to avoid spurious claims of coverage. Experiments on ill-conditioned test cases and the German Credit dataset demonstrate that, compared to Fisher-informed low-rank or diagonal preconditioning strategies, the proposed approach achieves 1.13–2.45× higher gradient-normalized effective sample size (ESS) at equivalent posterior quality.

This work addresses the limitations of conventional learning rate warmup strategies, which rely on heuristic hyperparameter tuning and lack theoretical grounding—particularly exhibiting instability under norm-constrained optimizers such as Muon and Lion. Building upon a generalized smoothness assumption that links local curvature to the suboptimality gap, the paper derives, for the first time, a learning rate schedule that naturally integrates both warmup and decay directly from convergence analysis. The resulting method is fully adaptive, requiring no additional hyperparameters and automatically adjusting warmup duration. Evaluated on LLaMA large language model pretraining, it consistently matches or surpasses the performance of manually tuned baselines across all experimental settings, significantly enhancing training efficiency and robustness.

This work addresses the barren plateau problem in quantum neural network training caused by poor parameter initialization by proposing a first-moment–based analytical framework. Combining operator concentration theory with numerical experiments, the study systematically evaluates and compares the efficacy of various initialization strategies—including identity, Gaussian, and several shifted or asymmetric distributions. For the first time, it establishes an operator-level criterion for initialization validity, demonstrating that viable initializations avoiding barren plateaus are highly non-unique and form exponentially many inequivalent families. Moreover, the research reveals that initializations with distinct first moments can converge to different local minima, indicating that intelligent initialization effectively transforms the exponential concentration challenge into a selection problem among numerous trainable regions.

This study addresses the challenge of real-time trajectory generation for on-orbit space manipulators approaching tumbling targets, where strong nonlinear coupling severely complicates control. The terminal approach is decomposed into two phases: center-of-mass translational planning and a subsequent attitude–manipulator torque allocation stage, with particular focus on the latter as the computational bottleneck. To accelerate this phase, the work innovatively introduces a causal Transformer as a warm-start strategy for sequential convex programming (SCP), integrating action chunking encoding and flow-matching decoding to simultaneously preserve trajectory optimality and significantly enhance computational efficiency and robustness. Experimental results across 300 test scenarios demonstrate up to a 28% reduction in SCP iterations and a 23% decrease in runtime; when applied to nonconvex feasibility projection, the method nearly halves computation time and effectively suppresses high-cost anomalous behaviors induced by heuristic initial guesses.