🤖 AI Summary
This work addresses optimization of gradient-dominated functions by proposing two first-order algorithms—Rescaled Gradient Flow (RGF) and Sign Gradient Flow (SGF)—derived from forward Euler discretizations of finite-time convergent gradient flows. It is the first to incorporate non-Lipschitz, discontinuous finite-time dynamical systems into first-order optimization frameworks, establishing rigorous finite-time local convergence guarantees under both deterministic and stochastic settings. To handle unknown or time-varying problem parameters, we design adaptive line search strategies and provide formal convergence analysis. Experiments on training deep networks over the SVHN dataset demonstrate that RGF and SGF significantly outperform standard optimizers in convergence speed, empirically validating their theoretical advantages and practical efficacy.
📝 Abstract
In this paper, we investigate the performance of two first-order optimization algorithms, obtained from forward Euler discretization of finite-time optimization flows. These flows are the rescaled-gradient flow (RGF) and the signed-gradient flow (SGF), and consist of non-Lipscthiz or discontinuous dynamical systems that converge locally in finite time to the minima of gradient-dominated functions. We propose an Euler discretization for these first-order finite-time flows, and provide convergence guarantees, in the deterministic and the stochastic setting. We then apply the proposed algorithms to academic examples, as well as deep neural networks training, where we empirically test their performances on the SVHN dataset. Our results show that our schemes demonstrate faster convergences against standard optimization alternatives.