🤖 AI Summary
Existing stochastic optimization theory is confined to Hilbert spaces, limiting its applicability to non-Euclidean geometries—such as simplices and probability manifolds—arising in mirror descent, natural gradient methods, and KL-regularized language model training.
Method: We propose the first unified stochastic optimization framework grounded in Banach spaces and Bregman geometry, eliminating reliance on inner products and supporting arbitrary smooth convex norms. Crucially, we introduce a novel over-relaxation parameter λ > 2, enabling the first derivation of Bregman–Fejér monotonicity and convergence guarantees in general Banach spaces.
Contribution/Results: Our framework unifies stochastic mirror descent, adaptive learning, and large-model training dynamics. Empirical evaluation on UCI datasets, Transformer, Actor-Critic, and distilGPT-2 demonstrates up to 20% faster convergence, significantly reduced variance, and improved accuracy—validating both theoretical generality and practical efficacy.
📝 Abstract
Stochastic optimization powers the scalability of modern artificial intelligence, spanning machine learning, deep learning, reinforcement learning, and large language model training. Yet, existing theory remains largely confined to Hilbert spaces, relying on inner-product frameworks and orthogonality. This paradigm fails to capture non-Euclidean settings, such as mirror descent on simplices, Bregman proximal methods for sparse learning, natural gradient descent in information geometry, or Kullback--Leibler-regularized language model training. Unlike Euclidean-based Hilbert-space methods, this approach embraces general Banach spaces. This work introduces a pioneering Banach--Bregman framework for stochastic iterations, establishing Bregman geometry as a foundation for next-generation optimization. It (i) provides a unified template via Bregman projections and Bregman--Fejer monotonicity, encompassing stochastic approximation, mirror descent, natural gradient, adaptive methods, and mirror-prox; (ii) establishes super-relaxations ($λ> 2$) in non-Hilbert settings, enabling flexible geometries and elucidating their acceleration effect; and (iii) delivers convergence theorems spanning almost-sure boundedness to geometric rates, validated on synthetic and real-world tasks. Empirical studies across machine learning (UCI benchmarks), deep learning (e.g., Transformer training), reinforcement learning (actor--critic), and large language models (WikiText-2 with distilGPT-2) show up to 20% faster convergence, reduced variance, and enhanced accuracy over classical baselines. These results position Banach--Bregman geometry as a cornerstone unifying optimization theory and practice across core AI paradigms.