A Contraction Framework for Stochastic Operators with Bootstrapping: Application to TD Learning

2026-09-24

AI Summary

This study addresses the limitations of existing convergence proofs for algorithms combining sampled updates with frozen target refreshment, which typically rely on specific structural assumptions and uniformly bounded errors. By modeling sampled updates as stochastic operators, this work proposes a general contraction analysis framework that requires no gradient structure assumptions and permits errors to grow across iterations. The framework unifies the convergence bounds of deterministic frozen targets and stochastic gradients, effectively relaxing restrictions on linear approximations and uniform error boundedness. Based on this formulation, finite-time bounds are derived for arbitrary target update intervals, and geometric convergence is established. Temporal difference experiments further validate the theoretically predicted contraction rates and the scaling behavior of error lower bounds.

Abstract

Many iterative algorithms rely on bootstrapping. A variable is updated using a second, frozen copy as a target, which is periodically replaced with the updated variable. Majorize-minimize and inexact proximal-point methods share this structure, as does temporal-difference (TD) learning. However, existing convergence guarantees for scenarios that combine sampled updates with targets refreshed only every $K$ steps rely on the specific structure of the update, such as linear approximation or gradient-based inner steps, and on uniformly bounded sampling error. We instead model the sampled update as a stochastic operator on the parameter space, which reduces the analysis to a contraction argument that needs no gradient structure and allows the sampling error to grow with the iterates. Within this framework, we derive a finite-time bound for i.i.d. samples and any target-update period $K$. We show that the iterates converge geometrically in root mean square to a ball around the fixed point, provided the sensitivity to the frozen target is smaller than the contraction slack of the inner map. Existing deterministic frozen-target contraction and stochastic-gradient-type bounds follow as special cases of our framework, and simulations of TD learning reproduce the predicted contraction rate and scaling of the error floor with the step size.

Problem

• bootstrapping
• stochastic operators
• contraction framework
• TD learning
• convergence guarantees

Innovation

• stochastic operators
• contraction framework
• bootstrapping
• TD learning
• finite-time bound