Quantile Multi-Armed Bandits with 1-bit Feedback

2025-02-10

AI Summary

This paper studies risk-sensitive best-arm identification under communication constraints: identifying the arm with the highest τ-quantile reward when only 1-bit feedback is available per pull. We establish, for the first time, a tight asymptotic sample complexity lower bound for this problem, showing that 1-bit feedback incurs negligible statistical efficiency loss—the minimal sampling cost differs from the unconstrained case by only a constant factor. Methodologically, we integrate noisy binary search, robust quantile estimation, and information-theoretic lower bound analysis to derive instance-dependent upper and lower bounds that match up to constant factors. Our key contributions are: (i) establishing the statistical feasibility of quantile-based arm identification under extreme bandwidth limitation; and (ii) characterizing the fundamental trade-off—demonstrating that risk sensitivity and communication efficiency are inherently compatible within a precise information-theoretic boundary.

Technology Category

• Machine Learning: Online Learning & Bandits
• Reasoning under Uncertainty: Stochastic Optimization
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Security and Privacy: Large-scale security measurements
• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems

Abstract

In this paper, we study a variant of best-arm identification involving elements of risk sensitivity and communication constraints. Specifically, the goal of the learner is to identify the arm with the highest quantile reward, while the communication from an agent (who observes rewards) and the learner (who chooses actions) is restricted to only one bit of feedback per arm pull. We propose an algorithm that utilizes noisy binary search as a subroutine, allowing the learner to estimate quantile rewards through 1-bit feedback. We derive an instance-dependent upper bound on the sample complexity of our algorithm and provide an algorithm-independent lower bound for specific instances, with the two matching to within logarithmic factors under mild conditions, or even to within constant factors in certain low error probability scaling regimes. The lower bound is applicable even in the absence of communication constraints, and thus we conclude that restricting to 1-bit feedback has a minimal impact on the scaling of the sample complexity.

Problem

• Identifies highest quantile reward arm
• Uses 1-bit feedback per arm pull
• Minimizes sample complexity impact

Innovation

• Quantile reward estimation
• Noisy binary search
• 1-bit feedback optimization