🤖 AI Summary
This work investigates the dynamical behavior of Nesterov-type accelerated gradient methods—including variable- and constant-momentum NAG and NCM—in escaping strict saddle points and converging to local minima for smooth nonconvex optimization. Employing non-asymptotic and asymptotic dynamical systems analysis, Lyapunov function construction, and manifold characterization near saddle points, we establish the first rigorous proof that variable-parameter NAG almost surely avoids strict saddle points. We introduce two asymptotic rate metrics and derive linear-scale estimates for saddle-point escape time. Moreover, we identify a subclass of accelerated methods that simultaneously achieves near-optimal convergence rates and strong escape capability. Our results provide a unified characterization of escape performance and local convergence rates across mainstream acceleration algorithms, establishing a novel theoretical framework for designing and analyzing accelerated methods in nonconvex optimization.
📝 Abstract
This paper considers the problem of understanding the behavior of a general class of accelerated gradient methods on smooth nonconvex functions. Motivated by some recent works that have proposed effective algorithms, based on Polyak's heavy ball method and the Nesterov accelerated gradient method, to achieve convergence to a local minimum of nonconvex functions, this work proposes a broad class of Nesterov-type accelerated methods and puts forth a rigorous study of these methods encompassing the escape from saddle-points and convergence to local minima through a both asymptotic and a non-asymptotic analysis. In the asymptotic regime, this paper answers an open question of whether Nesterov's accelerated gradient method (NAG) with variable momentum parameter avoids strict saddle points almost surely. This work also develops two metrics of asymptotic rate of convergence and divergence, and evaluates these two metrics for several popular standard accelerated methods such as the NAG, and Nesterov's accelerated gradient with constant momentum (NCM) near strict saddle points. In the local regime, this work provides an analysis that leads to the"linear"exit time estimates from strict saddle neighborhoods for trajectories of these accelerated methods as well the necessary conditions for the existence of such trajectories. Finally, this work studies a sub-class of accelerated methods that can converge in convex neighborhoods of nonconvex functions with a near optimal rate to a local minima and at the same time this sub-class offers superior saddle-escape behavior compared to that of NAG.