π€ AI Summary
This work addresses the lack of a unified framework in existing optimizer design, which often relies on heuristic modifications and struggles to balance stability and generalization. The authors propose the first systematic approach that integrates control theory with Riemannian geometry, modeling the optimization process as a discrete-time controlled dynamical system on a Riemannian manifold. By introducing normally attracting invariant manifolds (NAIMs) and strict Lyapunov functions, they establish a theoretically grounded framework for generating optimizers with provable convergence guarantees. This framework not only recovers classical algorithms but also yields novel optimizers that achieve state-of-the-art performance on large-scale benchmarks. Geometric diagnostics further validate the methodβs efficacy, offering a stable, interpretable, and theoretically rigorous toolkit for optimizer design.
π Abstract
We introduce Riemannian Lyapunov Optimizers (RLOs), a family of optimization algorithms that unifies classic optimizers within one geometric framework. Unlike heuristic improvements to existing optimizers, RLOs are systematically derived from a novel control-theoretic framework that reinterprets optimization as an extended state discrete-time controlled dynamical system on a Riemannian parameter manifold. Central to this framework is the identification of a Normally Attracting Invariant Manifold (NAIM), which organizes training dynamics into two distinct stages: rapid alignment of the speed state to a target graph, followed by controlled evolution within it. We formalize this by constructing a strict Lyapunov function that certifies convergence to a target manifold. This perspective yields a constructive ``optimizer generator"that not only recovers classic algorithms but enables the principled design of RLOs. We validate our theory via geometric diagnostics and demonstrate that grounding optimizer design in control theory yields state-of-the-art performance in large-scale benchmarks. Overall, RLOs bridge control theory and modern machine learning optimization, providing a unified language and a systematic toolkit for designing stable, effective optimizers.