🤖 AI Summary
This paper investigates the dynamical behavior of Thompson sampling under the joint asymptotic regime of small gaps—where arm mean differences scale as $O(sqrt{gamma})$—and long horizons—where the time horizon scales as $O(1/gamma)$. Using weak convergence analysis, we derive, for the first time from first principles, a diffusion approximation: we prove that the discrete-time update process converges weakly to an explicit stochastic differential equation (SDE) and its associated random ordinary differential equation (ODE). This limiting characterization unifies the asymptotics of diverse Thompson sampling variants—including those based on exponential families and bootstrap resampling—and reveals intrinsic robustness under model misspecification. Our results establish a universal limit theory for Thompson sampling in the small-gap regime and provide a novel analytical framework for continuous-time modeling and robustness analysis of bandit algorithms.
📝 Abstract
We study the behavior of Thompson sampling from the perspective of weak convergence. In the regime with small $gamma>0$, where the gaps between arm means scale as $sqrt{gamma}$ and over time horizons that scale as $1/gamma$, we show that the dynamics of Thompson sampling evolve according to discrete versions of SDE's and stochastic ODE's. As $gamma downarrow 0$, we show that the dynamics converge weakly to solutions of the corresponding SDE's and stochastic ODE's. Our weak convergence theory is developed from first principles using the Continuous Mapping Theorem, and can be easily adapted to analyze other sampling-based bandit algorithms. In this regime, we also show that the weak limits of the dynamics of many sampling-based algorithms -- including Thompson sampling designed for single-parameter exponential family rewards, and algorithms using bootstrap-based sampling to balance exploration and exploitation -- coincide with those of Gaussian Thompson sampling. Moreover, in this regime, these algorithms are generally robust to model mis-specification.