Permutation Equivariant Model-based Offline Reinforcement Learning for Auto-bidding

2025-06-22

AI Summary

Offline reinforcement learning (RL) for automated bidding faces two key challenges: limited state coverage in real-world datasets and environmental mismatch in simulator-generated data. To address these, we propose Model-based Reinforcement Learning for Bidding (MRLB), a novel offline RL framework that jointly trains policies using both real data and high-fidelity environment-model-synthesized data. Methodologically, MRLB introduces a permutation-equivariant neural network to model environment dynamics, ensuring symmetric generalization of state representations; it further designs a pessimism-constrained offline Q-learning algorithm that explicitly mitigates error propagation from model inaccuracies, thereby enhancing policy robustness. Evaluated on real-world advertising bidding tasks, MRLB substantially outperforms existing state-of-the-art methods—demonstrating significant improvements in policy performance, training stability, and cross-scenario generalization capability.

Technology Category

• Machine Learning: Online Learning & Bandits
• Game Theory and Economic Paradigms: Adversarial Learning
• Search and Optimization: Learning to Search

Application Category

• Economics, Online Markets and Human Computation: Uses of LLMs and GenAI for marketplace design, bidding, and strategic interactions
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: User modeling for targeted and personalized online advertising

Abstract

Reinforcement learning (RL) for auto-bidding has shifted from using simplistic offline simulators (Simulation-based RL Bidding, SRLB) to offline RL on fixed real datasets (Offline RL Bidding, ORLB). However, ORLB policies are limited by the dataset's state space coverage, offering modest gains. While SRLB expands state coverage, its simulator-reality gap risks misleading policies. This paper introduces Model-based RL Bidding (MRLB), which learns an environment model from real data to bridge this gap. MRLB trains policies using both real and model-generated data, expanding state coverage beyond ORLB. To ensure model reliability, we propose: 1) A permutation equivariant model architecture for better generalization, and 2) A robust offline Q-learning method that pessimistically penalizes model errors. These form the Permutation Equivariant Model-based Offline RL (PE-MORL) algorithm. Real-world experiments show that PE-MORL outperforms state-of-the-art auto-bidding methods.

Problem

• Bridges simulator-reality gap in auto-bidding RL
• Expands state coverage beyond fixed datasets
• Ensures model reliability with equivariant architecture

Innovation

• Model-based RL bridging simulator-reality gap
• Permutation equivariant model for generalization
• Pessimistic Q-learning penalizing model errors