second-order taylor expansion

Second-order optimization methods—e.g., Newton’s method—systematically fail in deep neural network training when applied with the exact Hessian, despite its theoretical role in capturing local curvature. Method: We combine nonlinear discrete-time dynamical modeling with exact Hessian analysis of regression loss landscapes to characterize optimization dynamics under discrete parameter updates. Contribution/Results: We reveal a geometric mismatch mechanism: critical points are not densely clustered near local minima but instead form a sparse, high-dimensional saddle-dominated structure. Consequently, exact curvature information induces directional misalignment and step-size instability—not convergence acceleration—during discrete updates. This challenges the long-standing “ubiquitous local minima” hypothesis and provides the first curvature-utilization mismatch explanation for second-order failure in deep learning. Our framework delivers both a novel theoretical perspective on optimization geometry and empirical validation, advancing the understanding of why canonical second-order methods underperform in modern neural network training.

This work investigates how training data shape the prediction mechanisms of neural networks through optimization trajectories, with a particular focus on higher-order effects under stochastic optimization and momentum. We introduce, for the first time, a second-order path kernel interpolation formula that expresses model predictions as an integral along the optimization path, where the leading term is weighted by the loss curvature and a correction term couples the covariance of gradient noise. This formulation naturally extends to momentum-based stochastic gradient descent. By leveraging path integrals, second-order Taylor expansions, and stochastic differential equation analysis, our framework precisely characterizes how stochasticity and momentum influence the interpolation structure and provides concentration bounds for the final prediction, quantifying the scale of predictive fluctuations.

This work addresses the degradation of convergence and robustness in the Levenberg-Marquardt (LM) method caused by parameterization-induced parameter-effect curvature, which leads to inconsistency between finite-step updates and linear model predictions. To resolve this, the authors propose RNC-LM, a novel approach that constructs high-order geometric updates using Riemannian normal coordinates. By iteratively eliminating the tangential components of residual acceleration within a moving tangent space, RNC-LM achieves, for the first time, high-order consistent correction of parameter-effect curvature at finite step sizes. This significantly enhances the geometric fidelity of the optimization trajectory, improving convergence and robustness on classical nonlinear least-squares problems. Empirically, RNC-LM reduces the relative L² error to 1e⁻³ in a reaction-diffusion physics-informed neural network (PINN) failure case and accelerates large-scale potential energy surface fitting by 34× compared to standard LM.

This work addresses optimization of gradient-dominated functions by proposing two first-order algorithms—Rescaled Gradient Flow (RGF) and Sign Gradient Flow (SGF)—derived from forward Euler discretizations of finite-time convergent gradient flows. It is the first to incorporate non-Lipschitz, discontinuous finite-time dynamical systems into first-order optimization frameworks, establishing rigorous finite-time local convergence guarantees under both deterministic and stochastic settings. To handle unknown or time-varying problem parameters, we design adaptive line search strategies and provide formal convergence analysis. Experiments on training deep networks over the SVHN dataset demonstrate that RGF and SGF significantly outperform standard optimizers in convergence speed, empirically validating their theoretical advantages and practical efficacy.

This work addresses the slow convergence of gradient descent on complex objectives and its reliance on strong global smoothness assumptions. We introduce *directional smoothness*, a novel geometric concept characterizing the local smoothness of the objective function along the optimization trajectory—thereby circumventing restrictive global Lipschitz continuity requirements. Leveraging this path-dependent characterization, we derive a trajectory-aware suboptimality bound and formulate an implicit adaptive step-size equation. We theoretically establish that Polyak’s step size and normalized gradient descent inherently achieve path-adaptive fast convergence. Our methodology integrates directional smoothness analysis, implicit step-size design, and convergence theory for both convex and nonconvex settings. Experiments on logistic regression demonstrate that our new bound substantially improves upon classical $L$-smoothness-based guarantees. Notably, this is the first work to provide path-dependent convergence rates for these two canonical algorithms without requiring prior knowledge of smoothness parameters.

This work addresses the lack of a unified modular framework for analyzing adaptive optimizers, which hinders a precise characterization of their behavior under constraints on directional reachability, information budgets, and update rules. We propose a geometric–non-geometric decoupled calculus for optimizers: the geometric module, constituted by a family of positive-definite cometrics, captures realizable descent directions, while the non-geometric module governs mechanisms such as information processing, memory, and control. Within this framework, we establish a direction expressivity theorem and a residual theory for constrained cometric families, disentangling directional expressiveness from condition-number complexity and recasting optimizer design as a Pareto optimization problem under modular budgets. Theoretically, we prove that fully positive-definite geometry exactly spans all strictly descending directions; experiments demonstrate that high-information full-metric probes attain numerical precision on deterministic quadratic problems, and a Muon-style implementation preliminarily validates the auditability of matrix-operator updates.

This work addresses the suboptimality of standard stochastic gradient descent (SGD) convergence analyses, which neglect the geometric heterogeneity of gradient noise in parameter space. To remedy this, the authors propose Curvature-Weighted Gradient Diversity (CWGD), a novel noise metric that weights sample-wise gradient diversity by the inverse square root of the Hessian, thereby aligning the noise characterization with the underlying optimization geometry. Building on this metric, they design the CWGD-Cosine learning rate schedule. Theoretical analysis demonstrates that this approach reduces the asymptotic optimization error to half that of standard cosine annealing. Empirical validation across varying condition numbers, batch sizes, and noise structures consistently shows approximately 20% lower final error on average, with negligible computational overhead.

This work addresses the poor adaptability to anisotropic curvature and sensitivity to affine transformations in unconstrained smooth optimization by proposing a novel algorithm grounded in affine differential geometry. The method introduces, for the first time in optimization, the equiaffine normal vector of level-set hypersurfaces to construct a volume-preserving, affine-invariant search direction that aligns with the Newton direction for strictly convex quadratic problems, thereby achieving one-step convergence. Under standard smoothness assumptions, the algorithm is globally convergent; it exhibits linear convergence under strong convexity or the Polyak–Łojasiewicz condition, and attains quadratic local convergence near nondegenerate minima. Numerical experiments confirm its robustness and efficiency under anisotropic scaling.

This work addresses the problem of smooth path tracing under simultaneous upper and lower curvature bounds by proposing a novel curvature-bounded geodesic model. Formulated within the Hamilton-Jacobi-Bellman (HJB) partial differential equation framework, the model introduces bilateral curvature constraints into the HJB formalism for the first time, enabling strong control over the geometric properties of generated paths. An efficient numerical discretization scheme is devised to balance path smoothness, rigidity, and elasticity. Experimental results demonstrate that the method robustly produces high-quality, curvature-constrained optimal paths in applications such as robotic motion planning and image curve structure tracking, significantly extending the capabilities of conventional single-bound constrained models.

This work addresses the computational and memory bottlenecks in accurately approximating the Hessian matrix in modern deep networks. The authors propose a novel approach that constructs a structured Hessian approximation from a single gradient by analytically averaging over the symmetry group actions on the weight space that leave the loss invariant. This approximation enables efficient estimation, storage, and inversion, and—by explicitly incorporating weight symmetries into curvature modeling—provides a unified perspective that subsumes existing methods such as Shampoo and Muon. Empirical evaluations demonstrate that the framework consistently enhances second-order optimization across diverse architectures and small-scale language models, offering controllable accuracy-computation trade-offs and extensibility to downstream tasks including uncertainty quantification and continual learning.