gradient flow analysis

This work systematically investigates information propagation, training dynamics, and macroscopic limiting behavior of neural networks from a dynamical systems perspective. To address these challenges, we propose an augmented Neural ODE framework that unifies input-output mappings across diverse architectures; integrate Lyapunov exponent analysis with mean-field limit theory to characterize gradient descent stability and implicit bias emergence in the overparameterized regime; and extend graph measure methods to establish convergence theory for heterogeneous neural networks under graph limits—first revealing their formal connection to Kuramoto-type synchronization models. Key contributions include: (i) rigorous classification of representable function classes for MLPs and Neural ODEs; (ii) a dynamical-systems explanation of SGD stability; and (iii) a novel mean-field analytical paradigm scalable to large-scale graph neural networks. These results provide foundational dynamical insights for interpretable and robust AI, particularly in generative modeling and gradient-based optimization.

This study addresses the high computational cost and poor safety controllability in training ultra-large AI models. Methodologically, it introduces a novel optimization paradigm grounded in mechanistic interpretability, pioneering the application of “circuit analysis” to model gradient descent trajectories. It structures the parameter space into functional subnetworks and designs a progressive curriculum learning strategy to dynamically regulate optimization paths within a controlled environment. Key contributions include: (1) establishing a formal mapping between gradient flow dynamics and circuit-like structural representations, enabling interpretable modeling of optimization behavior; and (2) leveraging structural priors to guide curriculum design, significantly accelerating convergence while suppressing the emergence of harmful behaviors. Experiments across multiple benchmark tasks demonstrate over 30% reduction in training cost alongside improved behavioral controllability, offering a principled pathway toward efficient and safe large-model training.

This study addresses the critical issue of model instability in software engineering optimization, which leads to substantial variability across repeated experiments and undermines both credibility and practical utility. Rather than treating instability as mere random noise, this work conceptualizes it as a quantifiable and manageable property that should be integrated into standard evaluation frameworks. By systematically modulating label usage, model complexity, and partition scoring strategies—combined with multi-objective optimization, causal intervention, data locality analysis, and model calibration—the proposed approach significantly enhances result consistency. Empirical evaluation demonstrates that the optimized configuration reduces the standard deviation of error by 22% on average and outperforms default settings in 119 out of 127 datasets, achieving a 4.8-fold improvement in result consistency.

The reliability of deep neural network interpretability methods is hindered by weak theoretical foundations and insufficient formalization. Method: This paper introduces a spectral-analytic perspective to systematically uncover a pervasive spectral bias in gradient-based attribution, establishing the first formal analytical framework that unifies probabilistic modeling with spectral theory. Building on this, it theoretically justifies key design choices—including squared gradients and input perturbations—and proposes a standardized perturbation scale mechanism and SpectralLens, a novel spectral-aware attribution aggregation method to enhance explanation consistency. Results: Extensive experiments validate the theoretical insights: SpectralLens significantly improves attribution stability and reliability across diverse architectures and datasets, yielding consistent, interpretable, and spectrally grounded feature attributions. The work provides both a rigorous spectral-theoretic foundation for explainable AI and a practical, deployable tool for robust model interpretation.

Model explanations often suffer from inconsistency and unreliability, undermining trust in interpretability methods. Method: We propose the Fixed-Point Interpretability (FPI) framework, which formally defines a “fixed-point explanation” as one satisfying minimality, stability, and faithfulness. Leveraging fixed-point theory and convergence analysis, FPI recursively evaluates the interaction between a model and an explainer until convergence, exposing latent model behaviors and explainer weaknesses. The framework is instantiated for diverse explainers—including feature attribution methods and sparse autoencoders—and systematically verifies their convergence conditions. Contribution/Results: We introduce the first theoretical paradigm that models explanation stability as a fixed-point problem, establishing a new evaluation benchmark for interpretability. Experiments provide quantitative stability metrics and canonical failure cases, demonstrating significant improvements in explanation reliability and trustworthiness across multiple architectures and datasets.

Current machine learning evaluation practices predominantly rely on surface-level performance metrics, often neglecting the internal mechanisms of models. This work proposes trustworthy interpretability as a central evaluation paradigm and, for the first time, systematically demonstrates that it satisfies core criteria from the philosophy of science—namely falsifiability, reproducibility, and predictive power. By constructing an evaluation framework that integrates causal analysis with mechanistic probing, the study delineates three functional pathways through which interpretability enables the identification of behavioral origins, detection of latent flaws, and prediction of potential failure modes. This approach advances model assessment beyond performance-oriented benchmarks toward a deeper understanding of underlying mechanisms.

Existing model explanation methods often suffer from attribution bias or even erroneous interpretations due to inadequate consideration of baseline selection. This work reformulates the model explanation task by unifying gradient-based methods, Integrated Gradients (IG), and Taylor expansion approaches, thereby systematically revealing— for the first time—the pivotal role of the baseline in attribution. Building on this insight, the authors propose an evaluation framework grounded in attribution error and develop a general-purpose explanation method with a well-defined, principled baseline that supports feature attribution at arbitrary network layers. The refined IG variant significantly improves explanation accuracy across multiple benchmarks, and attributions derived from different layers coherently reflect the hierarchical nature of feature extraction in deep networks.

This work addresses the problem of data deletion in deep learning—efficiently predicting how a model’s behavior changes after removing a subset of its training data. The authors propose a novel approach grounded in a stability assumption, introducing high-order complex directional derivatives into locally sketched arithmetic circuits for the first time and leveraging forward-mode automatic differentiation for efficient computation. Their method incurs only a poly(1/ε) overhead over standard training during a one-time precomputation phase, requires storage equivalent to poly(1/ε) model copies, and achieves prediction latency only poly(1/ε) times slower than standard inference. Crucially, the prediction error vanishes as ε approaches zero, and the framework is compatible with state-of-the-art, high-performance AI models.

Traditional linearization-based analyses often fail to accurately characterize the stability of optimization algorithms under nonlinear dynamics, leading to potential misjudgments. This work addresses this limitation by explicitly analyzing the nonlinear behavior of gradient descent (GD) and stochastic gradient descent (SGD) near minima. It proposes a multivariate GD stability criterion based on higher-order derivatives that captures stable oscillatory regimes beyond the reach of linear analysis. Furthermore, the study reveals that the overall stability of SGD can be dominated by a single unstable batch rather than governed by averaging effects. Theoretically, it is proven that when all batches are linearly stable, SGD is nonlinearly stable in expectation; however, the presence of even one unstable batch is sufficient to induce global divergence.

This study investigates how weight decay enhances training stability in deep learning through a unified framework combining dynamical systems analysis, the Edge of Stability (EoS) theory, the Neural Tangent Kernel (NTK) perspective, and mathematical modeling. The authors demonstrate that weight decay induces architecture-dependent phase transitions in both CNNs and MLPs, rooted in the global alignment between parameter vectors and curvature gradients. This mechanism effectively suppresses asymptotic sharpness and modulates oscillations in optimization trajectories. Furthermore, the work reveals that conventional curvature-based thresholds derived under convexity assumptions fail under regularization, thereby establishing weight decay as a nontrivial yet essential regulator of stable training dynamics.