Finite-Time Concentration and Convergence Rates for Projected Two-Time-Scale Stochastic Approximation with Markov Noise

2026-09-28

AI Summary

This study addresses the challenge of analyzing convergence rates in projected two-timescale stochastic approximation with Markovian noise, where discontinuous vector fields at domain boundaries complicate theoretical guarantees. Moving beyond standard Lipschitz assumptions, the proposed method leverages the Skorokhod reflection mapping to establish high-probability tracking bounds that decouple noise from bias, integrating Lyapunov stability theory with concentration inequalities to complete the convergence analysis. This work provides the first optimal joint convergence rates for boundary equilibria and extends them to fast-iterate projection settings. It achieves an almost-sure convergence rate of $O(n^{-1/2}\log n)$ and demonstrates a value-gap error of $O(n^{-1}\log n)$ in Actor-Critic applications.

Abstract

We study finite-time concentration and convergence rates for projected two-time-scale stochastic approximation driven by a controlled Markov chain. The averaged fast map is contractive, while the slow iterate is projected onto a compact convex polyhedron. The associated projected ordinary differential equation may have a discontinuous vector field at the boundary, preventing a direct application of standard analyses based on Lipschitz vector fields. Using the Skorokhod map, we establish explicit high-probability bounds for tracking the moving fast equilibrium and the projected slow dynamics. These bounds separate martingale fluctuations, Markov-noise residuals, and the bias due to time-scale separation. A Lipschitz Lyapunov function satisfying a uniform decrease condition over fixed time intervals yields almost-sure convergence, with explicit last-iterate rates when the decrease admits a power lower bound. Under uniform Lyapunov contraction, polynomial step sizes yield joint fast-tracking and slow Lyapunov-error exponents arbitrarily close to $1/3$. Under the additional assumption that the reduced slow update map is a Euclidean contraction, logarithmically separated step sizes improve the joint rate to $O(n^{-1/2}\log n)$ almost surely, including for boundary equilibria. The same rate holds under a distinct geometric condition involving a strictly attracting face of a box and a fast equilibrium that is constant on that face. An actor-critic application achieves an almost-sure value-gap rate of $O(n^{-1}\log n)$ relative to the optimum within the constrained policy class. Further applications include projected TD(0) and projected stochastic gradient descent. We also extend the analysis to projection of the fast recursion under Euclidean contractivity.

Problem

• Two-Time-Scale Stochastic Approximation
• Markov Noise
• Projected Dynamics
• Finite-Time Convergence
• Discontinuous Vector Field

Innovation

• Two-time-scale stochastic approximation
• Markov noise
• Skorokhod map
• Finite-time convergence rates
• Actor-critic