Clustered KL-barycenter design for policy evaluation

2025-03-04

AI Summary

This paper addresses the importance sampling (IS) evaluation of multi-objective policies in stochastic bandits, proposing a sample-efficient behavior policy design method. Existing approaches rely on a single KL-barycenter, failing to capture the heterogeneity among target policies. To address this, we introduce the Clustered KL-barycenter Policy Evaluation (CKL-PE) framework—the first to jointly cluster target policies and compute a Wasserstein-type KL-barycenter independently for each cluster, thereby more accurately modeling inter-policy divergence. Theoretically, CKL-PE achieves a tighter upper bound on sample complexity compared to single-barycenter methods. Empirically, it significantly reduces the variance of IS estimators, enhancing both evaluation stability and data efficiency. Extensive experiments demonstrate that CKL-PE consistently outperforms single-barycenter baselines and state-of-the-art alternatives across diverse multi-objective bandit settings.

Technology Category

• Machine Learning: Online Learning & Bandits
• Search and Optimization: Sampling/Simulation-based Search
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

In the context of stochastic bandit models, this article examines how to design sample-efficient behavior policies for the importance sampling evaluation of multiple target policies. From importance sampling theory, it is well established that sample efficiency is highly sensitive to the KL divergence between the target and importance sampling distributions. We first analyze a single behavior policy defined as the KL-barycenter of the target policies. Then, we refine this approach by clustering the target policies into groups with small KL divergences and assigning each cluster its own KL-barycenter as a behavior policy. This clustered KL-based policy evaluation (CKL-PE) algorithm provides a novel perspective on optimal policy selection. We prove upper bounds on the sample complexity of our method and demonstrate its effectiveness with numerical validation.

Problem

• Design sample-efficient behavior policies for policy evaluation
• Cluster target policies to minimize KL divergence
• Prove upper bounds on sample complexity of CKL-PE

Innovation

• Clustered KL-barycenter for policy evaluation
• Sample-efficient behavior policy design
• KL divergence-based policy clustering