Characterizing Bias in Post-Bandit Inference under Index Algorithms

2026-08-02

AI Summary

This work addresses the significant bias in sample means induced by multi-armed bandit algorithms—such as UCB1—in adaptive sampling, which severely compromises downstream statistical inference. The paper introduces the notion of an “effective exploration rate” and establishes, for the first time, a theoretical trade-off between this bias and cumulative regret. By employing empirical fluid approximations to characterize sampling dynamics and combining them with asymptotic analysis, the authors derive sharp leading-order expressions for both the sample mean bias and the associated Z-statistic. Their analysis reveals that under UCB1, the standardized bias for suboptimal arms decays extremely slowly—only at a rate of \(1/\sqrt{\log T}\)—and demonstrates that while enhanced exploration mitigates bias, it simultaneously increases regret, thereby providing a theoretical foundation for balancing statistical accuracy against learning efficiency.

Abstract

Bandit algorithms generate data for downstream inference, but adaptive sampling biases post-bandit sample means. We analyze this bias for stable index algorithms, including UCB1 and its generalizations, and derive sharp leading-order expressions for the sample-mean bias and expected $Z$-statistic. Our characterization reveals the algorithmic origin of bias through a key index-function-dependent quantity, which we term effective exploration rate. For example, under UCB1, the effective exploration rate is of order $\sqrt{\log T}$, and the standardized bias of any arm (that is not uniquely optimal) decays at the extremely slow rate $1/\sqrt{\log T}$. We also show how the choice of the index function affects both regret and bias, which reveals a regret-bias trade-off: more exploratory algorithm reduces bias but increases regret. Our sharp characterization for bias uses a novel empirical fluid approximation of the algorithm's sampling dynamics, which may be of independent interest.

Problem

• bandit algorithms
• adaptive sampling bias
• post-bandit inference
• index algorithms
• regret-bias trade-off

Innovation

• effective exploration rate
• post-bandit inference
• regret-bias trade-off
• empirical fluid approximation
• index algorithms