Concentration and Mean-Square Bounds for Contractive Stochastic Approximation: A Unified Elementary Approach

2026-07-20

AI Summary

This work addresses stochastic approximation problems involving multiplicative noise and unbounded iterates under compression, proposing a unified and elementary analytical framework. By directly constructing a first-order Lyapunov drift inequality for the error norm, the approach avoids sophisticated tools such as smoothing or Moreau envelopes. Combining averaged noise sequences with auxiliary iterates, the method employs inductive expectation arguments to derive mean-square error bounds and, for the first time under multiplicative noise, establishes a full-trajectory maximal concentration bound with sub-Gaussian tails via probabilistic induction and the Azuma–Hoeffding inequality. The stepsize schedule depends only logarithmically on the confidence parameter, eliminating the need for multi-stage warm-up procedures or smooth Lyapunov functions, thereby significantly simplifying the analysis. The framework is successfully applied to ℓ∞-contractive operators in reinforcement learning and exhibits strong generalizability.

Abstract

We establish mean-square and concentration bounds for stochastic approximation (SA) with arbitrary norm contractive mappings, under a multiplicative noise model where the noise may scale affinely with the norm of the iterates, and the iterates are potentially unbounded. These settings arise in reinforcement learning, where operators are often contractive in the $\ell_\infty$ norm and the noise scales with the iterates. To address the arbitrary norm, earlier works replace the non-smooth squared norm with a smooth Lyapunov function constructed via the generalized Moreau envelope. For concentration analysis, these works handle multiplicative noise and unbounded iterates through a multi-stage bootstrapping argument that starts from a time-varying worst-case bound and iteratively refines it. We instead present a unified and elementary analysis that yields both bounds. Using an averaged noise sequence and corresponding auxiliary iterates, we obtain a one-step Lyapunov drift inequality for the normed error directly, without smoothing the norm or constructing an envelope. For the mean-square bound, we combine this drift inequality with an induction argument showing that the iterates remain bounded in expectation. For the concentration bound, we develop a probabilistic induction over a sequence of "good" events on which the iterates are controlled, allowing the standard Azuma-Hoeffding bound to be applied. Our approach yields the first sub-Gaussian tailed maximal (all-time) concentration bound for SA under multiplicative noise, by allowing the stepsize to depend logarithmically on the confidence level. Beyond the specific setting considered here, we discuss the generalizability of these proof techniques to other noise models and iterative algorithms.

Problem

• stochastic approximation
• contractive mappings
• multiplicative noise
• concentration bounds
• mean-square bounds

Innovation

• stochastic approximation
• contractive mappings
• multiplicative noise
• concentration bounds
• Lyapunov drift