Refined PAC-Bayes Bounds for Offline Bandits

2025-02-17

AI Summary

This work improves PAC-Bayesian bounds for off-policy evaluation in offline bandits, focusing on empirical reward estimation. Building upon the framework of Seldin et al. (2010), we incorporate Rodríguez et al.’s (2024) novel event-space discretization and parameter optimization technique to derive two parameter-free PAC-Bayes bounds. Our method integrates Hoeffding–Azuma and Bernstein inequalities, enabling data-dependent optimal convergence rates “in probability” without requiring post-hoc access to observed data—a first in this setting. Theoretically, our bounds are strictly tighter than existing ones, achieving the same convergence rate as that attained under ideal posterior parameter selection. This yields significantly stronger theoretical guarantees and enhanced practical utility for offline policy evaluation.

Technology Category

Machine Learning: Online Learning & BanditsReasoning under Uncertainty: Stochastic OptimizationSearch and Optimization: Sampling/Simulation-based Search

Application Category

Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystemsUser Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendationGraph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs

Abstract

In this paper, we present refined probabilistic bounds on empirical reward estimates for off-policy learning in bandit problems. We build on the PAC-Bayesian bounds from Seldin et al. (2010) and improve on their results using a new parameter optimization approach introduced by Rodr'iguez et al. (2024). This technique is based on a discretization of the space of possible events to optimize the"in probability"parameter. We provide two parameter-free PAC-Bayes bounds, one based on Hoeffding-Azuma's inequality and the other based on Bernstein's inequality. We prove that our bounds are almost optimal as they recover the same rate as would be obtained by setting the"in probability"parameter after the realization of the data.

Problem

• Refine probabilistic bounds for off-policy learning.
• Optimize parameters using event space discretization.
• Provide parameter-free PAC-Bayes bounds for bandits.

Innovation

• Refined PAC-Bayes bounds
• Parameter optimization approach
• Discretization of event space