Satisficing Regret Minimization in Bandits

2024-06-10

AI Summary

This paper addresses the satisficing decision problem in multi-armed bandit optimization—selecting arms whose mean reward exceeds a given threshold with high frequency, rather than identifying the globally optimal arm. We formally define *satisficing regret*—the cumulative gap between realized rewards and the threshold—and prove it admits a constant upper bound. To achieve this, we propose the generic SELECT framework, which unifies adaptive sampling, lower-confidence-bound (LCB) hypothesis testing, and learning-oracle-driven iterative identification to handle both attainable and unattainable thresholds consistently. The framework attains constant satisficing regret across diverse bandit settings—including linear, contextual, and combinatorial environments—matching theoretical lower bounds. Empirical evaluations demonstrate substantial improvements over classical exploration algorithms.

Technology Category

• Machine Learning: Online Learning & Bandits
• Search and Optimization: Sampling/Simulation-based Search
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Motivated by the concept of satisficing in decision-making, we consider the problem of satisficing exploration in bandit optimization. In this setting, the learner aims at selecting satisficing arms (arms with mean reward exceeding a certain threshold value) as frequently as possible. The performance is measured by satisficing regret, which is the cumulative deficit of the chosen arm's mean reward compared to the threshold. We propose SELECT, a general algorithmic template for Satisficing REgret Minimization via SampLing and LowEr Confidence Bound Testing, that attains constant satisficing regret for a wide variety of bandit optimization problems in the realizable case (i.e., a satisficing arm exists). Specifically, given a class of bandit optimization problems and a corresponding learning oracle with sub-linear (standard) regret upper bound, SELECT iteratively makes use of the oracle to identify a potential satisficing arm with low regret. Then, it collects data samples from this arm, and continuously compares the LCB of the identified arm's mean reward against the threshold value to determine if it is a satisficing arm. As a complement, SELECT also enjoys the same (standard) regret guarantee as the oracle in the non-realizable case. Finally, we conduct numerical experiments to validate the performance of SELECT for several popular bandit optimization settings.

Problem

• Minimizing satisficing regret in bandit optimization problems
• Identifying arms with rewards exceeding a threshold value
• Developing an algorithm for constant regret in realizable cases

Innovation

• SELECT algorithm template for satisficing regret
• Uses sampling and lower confidence bound testing
• Attains constant regret with existing oracle integration