training pipeline design

This study investigates the stability of parameter directions during fine-tuning and their relationship to task transferability. Through spectral analysis, the authors find that the dominant singular vectors of pretrained weight matrices remain highly stable across fine-tuning and are shared across diverse tasks, suggesting that pretraining establishes a reusable spectral basis. Building on this observation, the work provides the first systematic evidence that these stable spectral directions encode transferable task structure and proposes an efficient fine-tuning method that optimizes only a small subset of spectral coefficients. Experiments on the GLUE benchmark demonstrate that updating as little as 0.2% of parameters yields competitive performance, confirming both the reusability of the spectral basis and the positive impact of pretraining scale on geometric transferability.

This work investigates the transfer mechanism in infinitely wide neural networks when both the source and downstream tasks operate in the feature-learning regime. We propose Elastic Weight Coupling (EWC), a unified framework modeling feature reuse across pretraining and fine-tuning. Within a Bayesian setting, we integrate gradient flow analysis with weight decay and the infinite-width limit to derive an adaptive feature kernel theory—whose structure depends explicitly on the data distributions and label geometries of both tasks. Our methodology encompasses posterior inference, gradient-flow dynamical modeling, and explicit kernel construction, supporting both linear and polynomial regression as well as validation on real-world datasets. Theoretically and empirically, we characterize the joint influence of coupling strength, feature-learning capacity, dataset scale, and task alignment on transfer performance—demonstrating consistent improvements in generalization across diverse scenarios.

This work uncovers the dynamical origin of low-dimensional spectral structure in weight matrices during neural network training and its connection to trainability. By modeling stochastic gradient descent (SGD) as the evolution of a random matrix ensemble, the study demonstrates that learning reshapes the spectral distribution and amplifies signal components, thereby inducing a Baik–Ben Arous–Péché (BBP) phase transition—a phenomenon newly introduced into neural network dynamics. Leveraging a linear teacher–student model combined with random matrix theory, the authors construct a trainability phase diagram governed by learning rate and initial weight variance, which is further extended to nonlinear and stochastic settings. Theoretical analysis yields precise spectral evolution laws, while numerical experiments on realistic architectures confirm the robust emergence of spectral alignment, establishing a unified spectral dynamical framework linking optimization hyperparameters to representation learning.

This work investigates how neural network width governs training dynamics. For single-hidden-layer linear networks, we derive the first exact analytical solution of learning dynamics at arbitrary finite width, unifying the characterization of the two-phase evolution—kernel learning and feature learning—and establishing a complete phase diagram parameterized by width, layer-wise learning rates, and initialization scale. Methodologically, we integrate analytical dynamical systems analysis, phase-diagram modeling, and empirical validation on nonlinear networks. Crucially, we identify three novel mechanisms operative during the feature-learning phase: alignment learning, de-alignment learning, and rescaling learning—each transcending the conventional kernel-method paradigm. These theoretical insights are empirically reproduced in realistic deep networks, offering a new conceptual framework for understanding training dynamics and designing adaptive optimization algorithms. (138 words)

This work investigates the intrinsic relationship between SGD training dynamics and the spectral structure of empirical Hessian and gradient matrices in high-dimensional multiclass classification. Methodologically, it employs theoretical modeling and spectral analysis to characterize the evolution of these matrices throughout training. The key contribution is the first rigorous proof that, in deep multilayer networks, SGD trajectories align layerwise with the anomalous eigensubspaces—i.e., those spanned by top eigenvalues—of both the layer-wise Hessian and gradient matrices. Crucially, the rank of the final-layer anomalous subspace monotonically degenerates during optimization and becomes severely deficient upon convergence to a suboptimal classifier, serving as a spectral diagnostic for suboptimal convergence in overparameterized networks. This result is formally established for high-dimensional mixture models and single-/two-layer neural networks. Moreover, the work quantifies the direct link between the evolution of the final-layer anomalous subspace and classification accuracy, offering a novel spectral-geometric perspective on deep learning optimization dynamics.

Despite hundreds of millions of parameters, transformer training trajectories evolve within only a few coherent directions. We introduce \emph{Spectral Edge Dynamics} (SED) to measure this structure: rolling-window SVD of parameter updates reveals a sharp boundary -- the \emph{spectral edge} -- between coherent optimization directions and stochastic noise, identified by the maximum consecutive singular value ratio $σ_k/σ_{k+1}$. Across a 51M-parameter TinyStories model (4~seeds) and GPT-2 124M under a distribution shift, the spectral edge exhibits a universal three-phase pattern (rise, plateau, collapse), signal rank adjusts with task complexity ($k^* = 2$ at 51M, $k^* = 3$ at 124M), and the directional coupling between spectral geometry and validation loss reverses with window size -- a \emph{lag flip} reflecting the timescale of trajectory integration. Johnson--Lindenstrauss projection to $d = 10W$ dimensions (e.g., $d = 100$ for $W = 10$) preserves the spectral gap within 5.7\%, making the framework applicable to models of arbitrary size. In companion work, the same spectral geometry provides early-warning signals of grokking -- predicting generalization 600--1{,}700 steps before it occurs across modular arithmetic, Dyck languages, and the SCAN benchmark.

This work addresses the limitations of mutual information–based feature learning in data-scarce settings, where noisy distribution estimation and the sensitivity of the H-Score to input basis rotations hinder performance. To mitigate these issues, the authors propose using the fast Fourier transform (FFT) as a low-overhead, data-agnostic unitary preconditioner that concentrates predictive dependencies into a few dominant spectral modes, thereby reducing truncation error in finite-width networks. They further introduce two training-free metrics—spectral entropy and cumulative dependency energy—to predict the efficacy of preprocessing and guide basis selection. Experiments across eight multivariate datasets demonstrate that FFT preconditioning can reduce normalized mean squared error (NMSE) by up to 50% under resource-constrained conditions, and the proposed metrics reliably anticipate both performance gains and failure cases.

This work addresses the challenge of modeling and forecasting stochastic nonlinear dynamical systems under noisy and partially observable conditions by proposing a deep spectral learning framework. The method employs a learnable neural encoder to construct Markovian latent states in a feature space, whose dynamics and observations are governed by learned transition and observation operators. It uniquely unifies spectral learning, Bayesian filtering, and Koopman mode decomposition within an end-to-end trainable architecture. Efficient state estimation is achieved through functional canonical correlation analysis, Galerkin projection, and a closed-form ridge-regularized solution. Experimental results demonstrate that the proposed approach significantly outperforms baseline methods—including sequential Bayesian filters and dynamic mode decomposition—across diverse scenarios, exhibiting strong robustness to both observation noise and partial observability.

This work addresses the exponential error growth in long-term predictions by neural operators for time-dependent partial differential equations, which arises from one-step Lipschitz constants. To mitigate this, the authors propose SpectraNet, an autoregressive neural operator that integrates truncated spectral convolutions with a U-Net multiscale architecture. By parameterizing residual target spectral blocks, the method reduces error accumulation from exponential to linear drift and introduces a semigroup consistency loss to decouple model parameters from spatial grid resolution. Evaluated on PDEs such as Navier–Stokes, SpectraNet achieves a relative L² error of 0.0822 with only 2.04M parameters—20% lower error and 2.33× fewer parameters than the baseline—and further improves to 0.0724 at 128² resolution. The model remains stable over long rollouts up to T=100 and requires less than 200ms for CPU inference.

Function-parameterized neural networks are highly sensitive to initialization, and conventional data-agnostic initialization schemes often fail to capture the structural characteristics of target signals, leading to slow convergence and unstable performance. This work proposes a prior-guided initialization strategy that, for the first time, integrates data-driven spectral priors into both network initialization and architecture design. Specifically, fast Fourier transform (FFT) is employed to extract seasonal priors that inform model depth and initial state, while residual regression is used to parameterize trend components. Without altering the training procedure, the proposed method significantly accelerates convergence, reduces performance variance, and improves computational efficiency across both synthetic and real-world datasets. Notably, it maintains reconstruction accuracy even when using a lower-dimensional encoder, consistently outperforming standard initialization approaches.