Logarithmic Smoothing for Adaptive PAC-Bayesian Off-Policy Learning

2025-06-12

AI Summary

This paper addresses the challenge of data-efficient learning in adaptive off-policy reinforcement learning, where the behavior policy dynamically evolves across iterations. We propose the first offline RL framework tailored for multi-round policy iteration and re-deployment scenarios. Methodologically, we extend PAC-Bayesian theory and logarithmic smoothing to the adaptive setting, designing a scalable Logarithmic Smoothing (LS) estimator adjustment mechanism that ensures bounded policy evaluation bias online and achieves faster convergence—both theoretically guaranteed. Compared to static offline methods, our framework attains comparable performance on standard benchmarks; under practical settings permitting intermediate policy deployment, it significantly outperforms existing adaptive approaches. Extensive multi-task experiments demonstrate its effectiveness, robustness, and generalization capability.

Technology Category

• Machine Learning: Online Learning & Bandits
• Search and Optimization: Learning to Search
• Multiagent Systems: Multiagent Learning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web

Abstract

Off-policy learning serves as the primary framework for learning optimal policies from logged interactions collected under a static behavior policy. In this work, we investigate the more practical and flexible setting of adaptive off-policy learning, where policies are iteratively refined and re-deployed to collect higher-quality data. Building on the success of PAC-Bayesian learning with Logarithmic Smoothing (LS) in static settings, we extend this framework to the adaptive scenario using tools from online PAC-Bayesian theory. Furthermore, we demonstrate that a principled adjustment to the LS estimator naturally accommodates multiple rounds of deployment and yields faster convergence rates under mild conditions. Our method matches the performance of leading offline approaches in static settings, and significantly outperforms them when intermediate policy deployments are allowed. Empirical evaluations across diverse scenarios highlight both the advantages of adaptive data collection and the strength of the PAC-Bayesian formulation.

Problem

• Extends PAC-Bayesian learning to adaptive off-policy scenarios
• Improves convergence rates with iterative policy refinement
• Outperforms static methods in adaptive data collection

Innovation

• Extends PAC-Bayesian learning to adaptive scenarios
• Adjusts Logarithmic Smoothing for multiple deployments
• Improves convergence rates with mild conditions