SAD Neural Networks: Divergent Gradient Flows and Asymptotic Optimality via o-minimal Structures

2025-05-14

AI Summary

This work investigates the gradient flow dynamics of fully connected feedforward neural networks with smooth activation functions (e.g., Sigmoid, tanh, softplus, GELU). To characterize the geometric structure of the loss landscape, we introduce the o-minimal structure theory to rigorously capture its semialgebraic properties, and integrate gradient flow analysis with tools from real algebraic geometry. We establish a precise equivalence between gradient flow divergence and asymptotic achievability of zero loss. Theoretically, we prove that—under any sufficiently favorable initialization—the gradient flow diverges to infinity in parameter space, yet the loss converges to the global minimum; moreover, there exists a neighborhood of initializations guaranteeing such divergence. This “divergent-yet-optimal” dichotomy is empirically validated via numerical ODE simulations and experiments on real-world datasets, revealing a nontrivial convergence mechanism intrinsic to overparameterized neural network optimization.

Technology Category

• Machine Learning: Optimization
• Search and Optimization: Non-convex Optimization
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data
• Social Networks and Social Media: Influence propagation, information diffusion, and the prediction on networks

Abstract

We study gradient flows for loss landscapes of fully connected feed forward neural networks with commonly used continuously differentiable activation functions such as the logistic, hyperbolic tangent, softplus or GELU function. We prove that the gradient flow either converges to a critical point or diverges to infinity while the loss converges to an asymptotic critical value. Moreover, we prove the existence of a threshold $varepsilon>0$ such that the loss value of any gradient flow initialized at most $varepsilon$ above the optimal level converges to it. For polynomial target functions and sufficiently big architecture and data set, we prove that the optimal loss value is zero and can only be realized asymptotically. From this setting, we deduce our main result that any gradient flow with sufficiently good initialization diverges to infinity. Our proof heavily relies on the geometry of o-minimal structures. We confirm these theoretical findings with numerical experiments and extend our investigation to real-world scenarios, where we observe an analogous behavior.

Problem

• Analyzes gradient flow convergence or divergence in neural networks
• Identifies threshold for loss convergence near optimal levels
• Proves asymptotic divergence for well-initialized flows in polynomial settings

Innovation

• Gradient flow analysis for neural networks
• Threshold-based loss convergence proof
• Divergence to infinity with good initialization