Nonlinear discretizations and Newton's method: characterizing stationary points of regression objectives

2025-10-13

AI Summary

Second-order optimization methods—e.g., Newton’s method—systematically fail in deep neural network training when applied with the exact Hessian, despite its theoretical role in capturing local curvature. Method: We combine nonlinear discrete-time dynamical modeling with exact Hessian analysis of regression loss landscapes to characterize optimization dynamics under discrete parameter updates. Contribution/Results: We reveal a geometric mismatch mechanism: critical points are not densely clustered near local minima but instead form a sparse, high-dimensional saddle-dominated structure. Consequently, exact curvature information induces directional misalignment and step-size instability—not convergence acceleration—during discrete updates. This challenges the long-standing “ubiquitous local minima” hypothesis and provides the first curvature-utilization mismatch explanation for second-order failure in deep learning. Our framework delivers both a novel theoretical perspective on optimization geometry and empirical validation, advancing the understanding of why canonical second-order methods underperform in modern neural network training.

Technology Category

• Search and Optimization: Non-convex Optimization
• Machine Learning: Optimization
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation

Abstract

Second-order methods are emerging as promising alternatives to standard first-order optimizers such as gradient descent and ADAM for training neural networks. Though the advantages of including curvature information in computing optimization steps have been celebrated in the scientific machine learning literature, the only second-order methods that have been studied are quasi-Newton, meaning that the Hessian matrix of the objective function is approximated. Though one would expect only to gain from using the true Hessian in place of its approximation, we show that neural network training reliably fails when relying on exact curvature information. The failure modes provide insight both into the geometry of nonlinear discretizations as well as the distribution of stationary points in the loss landscape, leading us to question the conventional wisdom that the loss landscape is replete with local minima.

Problem

• Investigating failure of neural network training with exact Hessian methods
• Analyzing geometry of nonlinear discretizations in optimization landscapes
• Challenging conventional wisdom about local minima distribution

Innovation

• Using exact Hessian matrix for neural network training
• Analyzing failure modes of second-order optimization methods
• Questioning abundance of local minima in loss landscape