Off-Policy Learning in Large Action Spaces: Optimization Matters More Than Estimation

2025-09-03

AI Summary

This paper identifies a critical limitation in offline contextual bandits: existing off-policy learning (OPL) methods prioritize statistical accuracy of off-policy evaluation (OPE) estimators while overlooking the optimization difficulty of the objective function under large action spaces—a key bottleneck for policy performance. To address this, we propose and theoretically justify a paradigm shift: using a simple weighted log-likelihood objective—despite sacrificing some estimation accuracy—yields superior optimization properties (e.g., gradient stability and convergence guarantees) compared to complex OPE-based objectives. Through rigorous theoretical analysis and systematic experiments in high-dimensional action spaces, we demonstrate that our approach consistently outperforms state-of-the-art OPL algorithms relying on high-accuracy OPE estimators. Our results establish “optimizability” as a more fundamental design criterion than “estimation accuracy,” introducing a new principle for OPL algorithm design.

Technology Category

• Machine Learning: Online Learning & Bandits
• Search and Optimization: Learning to Search
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web

Abstract

Off-policy evaluation (OPE) and off-policy learning (OPL) are foundational for decision-making in offline contextual bandits. Recent advances in OPL primarily optimize OPE estimators with improved statistical properties, assuming that better estimators inherently yield superior policies. Although theoretically justified, we argue this estimator-centric approach neglects a critical practical obstacle: challenging optimization landscapes. In this paper, we provide theoretical insights and extensive empirical evidence showing that current OPL methods encounter severe optimization issues, particularly as action spaces become large. We demonstrate that simpler weighted log-likelihood objectives enjoy substantially better optimization properties and still recover competitive, often superior, learned policies. Our findings emphasize the necessity of explicitly addressing optimization considerations in the development of OPL algorithms for large action spaces.

Problem

• Optimization challenges in off-policy learning with large action spaces
• Current methods face severe optimization issues as actions increase
• Simpler objectives needed for better optimization and superior policies

Innovation

• Weighted log-likelihood objectives for optimization
• Addressing challenging optimization landscapes explicitly
• Simpler objectives recover superior policies large actions