🤖 AI Summary
This work addresses policy evaluation in reinforcement learning under rare-event dynamics, where long-horizon dependencies and stringent relative accuracy requirements severely limit the efficiency of conventional Monte Carlo (MC) methods. Focusing on finite-state Markov chains, we adopt the least-squares temporal difference (LSTD) algorithm for policy evaluation and establish, for the first time, a central limit theorem for the LSTD estimator, along with an upper bound on its asymptotic relative variance. Theoretically, we prove that, in rare-event regimes, LSTD achieves a prescribed relative accuracy using only polynomially many state transitions, whereas MC requires exponentially many samples. This result provides the first rigorous asymptotic guarantee demonstrating the statistical superiority of temporal-difference–type methods over MC in long-horizon rare-event settings—breaking the longstanding reliance of prior analyses on frequent-event assumptions.
📝 Abstract
We quantify the efficiency of temporal difference (TD) learning over the direct, or Monte Carlo (MC), estimator for policy evaluation in reinforcement learning, with an emphasis on estimation of quantities related to rare events. Policy evaluation is complicated in the rare event setting by the long timescale of the event and by the need for emph{relative accuracy} in estimates of very small values. Specifically, we focus on least-squares TD (LSTD) prediction for finite state Markov chains, and show that LSTD can achieve relative accuracy far more efficiently than MC. We prove a central limit theorem for the LSTD estimator and upper bound the emph{relative asymptotic variance} by simple quantities characterizing the connectivity of states relative to the transition probabilities between them. Using this bound, we show that, even when both the timescale of the rare event and the relative accuracy of the MC estimator are exponentially large in the number of states, LSTD maintains a fixed level of relative accuracy with a total number of observed transitions of the Markov chain that is only emph{polynomially} large in the number of states.