UCB for Large-Scale Pure Exploration: Beyond Sub-Gaussianity

2025-11-27

AI Summary

Traditional UCB algorithms for large-scale pure exploration rely on sub-Gaussian assumptions and fail under heavy-tailed or non-sub-Gaussian reward distributions. Method: We propose Meta-UCB, a distribution-free framework requiring only bounded q-th moments (q > 3) of arm rewards—no prior knowledge of distribution type is needed. It leverages a location-scale structure and a maximum-sample-size stopping rule to enable theoretical analysis without distributional assumptions. Contribution/Results: This work establishes, for the first time, sample optimality of UCB-type algorithms in large-scale non-sub-Gaussian settings. We derive a distribution-independent lower bound on the probability of correct selection. Extensive numerical experiments demonstrate Meta-UCB’s superior robustness and efficacy under heavy-tailed reward distributions, significantly outperforming classical UCB variants.

Technology Category

• Reasoning under Uncertainty: Stochastic Optimization
• Machine Learning: Online Learning & Bandits
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web

Abstract

Selecting the best alternative from a finite set represents a broad class of pure exploration problems. Traditional approaches to pure exploration have predominantly relied on Gaussian or sub-Gaussian assumptions on the performance distributions of all alternatives, which limit their applicability to non-sub-Gaussian especially heavy-tailed problems. The need to move beyond sub-Gaussianity may become even more critical in large-scale problems, which tend to be especially sensitive to distributional specifications. In this paper, motivated by the widespread use of upper confidence bound (UCB) algorithms in pure exploration and beyond, we investigate their performance in the large-scale, non-sub-Gaussian settings. We consider the simplest category of UCB algorithms, where the UCB value for each alternative is defined as the sample mean plus an exploration bonus that depends only on its own sample size. We abstract this into a meta-UCB algorithm and propose letting it select the alternative with the largest sample size as the best upon stopping. For this meta-UCB algorithm, we first derive a distribution-free lower bound on the probability of correct selection. Building on this bound, we analyze two general non-sub-Gaussian scenarios: (1) all alternatives follow a common location-scale structure and have bounded variance; and (2) when such a structure does not hold, each alternative has a bounded absolute moment of order $q > 3$. In both settings, we show that the meta-UCB algorithm and therefore a broad class of UCB algorithms can achieve the sample optimality. These results demonstrate the applicability of UCB algorithms for solving large-scale pure exploration problems with non-sub-Gaussian distributions. Numerical experiments support our results and provide additional insights into the comparative behaviors of UCB algorithms within and beyond our meta-UCB framework.

Problem

• Extends UCB algorithms to non-sub-Gaussian distributions
• Addresses large-scale pure exploration with heavy-tailed data
• Ensures sample optimality in non-Gaussian bounded variance settings

Innovation

• Meta-UCB algorithm for non-sub-Gaussian distributions
• Distribution-free lower bound for correct selection probability
• Sample optimality in large-scale pure exploration problems