Non-Asymptotic Convergence of Stochastic Iterative Algorithms: A Lyapunov Framework

2026-05-29

AI Summary

This work addresses the finite-time convergence of stochastic iterative algorithms for fixed-point equations accessible only through a noisy oracle. The authors propose a norm-independent, unified Lyapunov function framework constructed via a generalized Moreau envelope, which integrates Lyapunov stability theory with stochastic approximation analysis. This framework accommodates complex settings such as Markovian noise, seminorm contractive operators, and dissipative operators, yielding sharp non-asymptotic convergence bounds in both high-probability and mean-square senses. As a result, it provides a unified and refined finite-time convergence guarantee for a broad class of algorithms, including stochastic gradient descent, linear stochastic approximation, Q-learning, and temporal difference learning.

Abstract

We survey Lyapunov-based techniques for the finite-time analysis of stochastic iterative algorithms, also known as stochastic approximation (SA) algorithms, for solving fixed-point equations $\bar{F}(x)=x$, where the operator $\bar{F}(\cdot)$ can only be accessed through a noisy oracle. We first focus on the standard setting in which $\bar{F}(\cdot)$ is contractive with respect to some norm and the noise is i.i.d., and explain how generalized Moreau envelopes serve as universal Lyapunov functions, regardless of the underlying norm. We then show how this framework yields mean-square convergence guarantees and applies to stochastic gradient descent, linear SA, and value-based reinforcement learning algorithms such as Q-learning and temporal-difference learning. Finally, we discuss extensions to Markovian noise, seminorm-contractive operators, dissipative operators, and high-probability bounds, and conclude with open problems. The goal is to present a unified and self-contained roadmap for the finite-time analysis of SA and its applications, especially in reinforcement learning.

Problem

• stochastic approximation
• non-asymptotic convergence
• fixed-point equations
• Lyapunov functions
• finite-time analysis

Innovation

• Lyapunov framework
• non-asymptotic convergence
• stochastic approximation
• Moreau envelope
• reinforcement learning