Technology Category
Natural Language Processing: Learning & Optimization for NLPSearch and Optimization: Learning to SearchComputer Vision: Learning & Optimization for CV
LyAm: Robust Non-Convex Optimization for Stable Learning in Noisy Environments
🤖 AI Summary
To address gradient instability and poor convergence of deep neural networks under noisy conditions, this paper proposes Lyapunov-Adam: the first adaptive optimizer integrating Lyapunov stability theory into optimization. It constructs a learnable Lyapunov function to dynamically modulate learning rates, providing rigorous convergence guarantees for non-convex optimization and strong robustness to noise. The method synergistically combines Adam’s momentum estimation with stability-driven step-size control and establishes a comprehensive theoretical analysis framework. On benchmark tasks including CIFAR-10 and CIFAR-100, Lyapunov-Adam outperforms state-of-the-art optimizers (e.g., AdamW, RMSProp, Lion), achieving significant improvements in test accuracy (+0.8%–1.3%), convergence speed (15%–22% acceleration), and training stability (37% reduction in gradient variance).
Technology Category
Application Category
📝 Abstract
Training deep neural networks, particularly in computer vision tasks, often suffers from noisy gradients and unstable convergence, which hinder performance and generalization. In this paper, we propose LyAm, a novel optimizer that integrates Adam's adaptive moment estimation with Lyapunov-based stability mechanisms. LyAm dynamically adjusts the learning rate using Lyapunov stability theory to enhance convergence robustness and mitigate training noise. We provide a rigorous theoretical framework proving the convergence guarantees of LyAm in complex, non-convex settings. Extensive experiments on like as CIFAR-10 and CIFAR-100 show that LyAm consistently outperforms state-of-the-art optimizers in terms of accuracy, convergence speed, and stability, establishing it as a strong candidate for robust deep learning optimization.
Problem
Research questions and friction points this paper is trying to address.
Addresses noisy gradients in deep neural network training
Enhances convergence stability in non-convex optimization
Improves accuracy and speed in noisy computer vision tasks
Innovation
Methods, ideas, or system contributions that make the work stand out.
LyAm integrates Adam with Lyapunov stability
Dynamic learning rate adjustment via Lyapunov theory
Proven convergence in non-convex optimization settings
🔎 Similar Papers
No similar papers found.
