Asymptotic regularity of a generalised stochastic Halpern scheme with applications

2024-11-07

arXiv.org

AI Summary

This work investigates asymptotic regularity of generalized stochastic Halpern-type iterations, unifying Halpern iteration, Tikhonov-regularized Krasnoselskii–Mann iteration, and other optimization algorithms within a fully abstract stochastic framework. We propose the first highly unified abstract convergence rate analysis framework: achieving optimal linear convergence in normed linear spaces and further attaining quadratic convergence in inner product spaces. We establish, for the first time, an abstract oracle complexity upper bound compatible with variance constraints. Our theoretical results are instantiated to reinforcement learning, yielding the first stochastic Q-learning algorithm framework with rigorous convergence guarantees. The core contributions constitute a threefold breakthrough—unprecedented abstraction, unification across algorithmic families, and optimality of convergence rates—thereby advancing the theoretical foundations of stochastic fixed-point and optimization methods.

Technology Category

• Reasoning under Uncertainty: Stochastic Optimization
• Machine Learning: Online Learning & Bandits
• Search and Optimization: Non-convex Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Economics, Online Markets and Human Computation: Social networks and social learning
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web

Abstract

We provide abstract, general and highly uniform rates of asymptotic regularity for a generalized stochastic Halpern-style iteration, which incorporates a second mapping in the style of a Krasnoselskii-Mann iteration. This iteration is general in two ways: First, it incorporates stochasticity in a completely abstract way rather than fixing a sampling method; secondly, it includes as special cases stochastic versions of various schemes from the optimization literature, including Halpern's iteration as well as a Krasnoselskii-Mann iteration with Tikhonov regularization terms in the sense of Boc{t}, Csetnek and Meier. For these particular cases, we in particular obtain linear rates of asymptotic regularity, matching (or improving) the currently best known rates for these iterations in stochastic optimization, and quadratic rates of asymptotic regularity are obtained in the context of inner product spaces for the general iteration. We utilize these rates to give bounds on the oracle complexity of such iterations under suitable variance assumptions and batching strategies, again presented in an abstract style. Finally, we sketch how the schemes presented here can be instantiated in the context of reinforcement learning to yield novel methods for Q-learning.

Problem

• Generalized stochastic Halpern iteration with abstract stochasticity
• Unified asymptotic regularity rates for optimization schemes
• Applications in reinforcement learning for Q-learning methods

Innovation

• Generalized stochastic Halpern-style iteration
• Incorporates Krasnoselskii-Mann iteration mapping
• Abstract stochasticity without fixed sampling