policy optimization

This paper addresses the limitation of the GRPO algorithm—its restriction to on-policy training—by systematically proposing and validating its first off-policy variant. Methodologically, we introduce a clipped surrogate objective function, adapt GRPO to the off-policy setting within the PPO framework, employ offline advantage estimation, and incorporate verifiable reward evaluation. We theoretically prove that this objective guarantees monotonic improvement in expected reward. Our key contributions are threefold: (1) the first successful extension of GRPO to the off-policy paradigm; (2) theoretical analysis demonstrating superior training stability and higher sample and memory efficiency compared to on-policy GRPO; and (3) empirical validation showing that off-policy GRPO matches or significantly outperforms the original on-policy version across multiple benchmark tasks.

This work investigates how self-supervised objectives can enhance an agent’s exploration and adaptation capabilities in out-of-distribution tasks during unsupervised pre-training. To this end, the authors propose ULEE, a method that integrates in-context reinforcement learning, adversarial goal generation, and performance-estimation-driven dynamic curriculum scheduling within a meta-learning framework to enable frontier-aware adaptive curriculum learning. Experiments on the XLand-MiniGrid benchmark demonstrate that ULEE substantially outperforms training from scratch, DIAYN, and other curriculum-based approaches, exhibiting superior zero-shot and few-shot generalization across novel goals, environmental dynamics, and map structures. Furthermore, ULEE provides high-quality initial policies that facilitate effective downstream fine-tuning.

Off-policy evaluation (OPE) for multiple target policies suffers from low sample efficiency, high variance, and the need for repeated data collection. Method: This paper proposes a unified off-policy evaluation framework based on a shared behavior policy. Contribution/Results: We theoretically establish, for the first time, the existence of a single customized behavior policy enabling consistent, unbiased estimation of multiple target policies using significantly fewer samples than individual on-policy rollouts. Building on this insight, we develop an optimization method that jointly minimizes estimation variance via importance sampling and derives a low-variance estimator supporting simultaneous multi-policy evaluation. Experiments across standard benchmarks demonstrate substantial improvements: average variance reduction of 35%–62% over state-of-the-art methods and 2–5× higher sample efficiency, achieving new state-of-the-art performance.

This work addresses the challenge of simultaneously ensuring behavioral preference satisfaction and monotonic policy improvement while maintaining efficient exploration in policy optimization. We propose the ε-retraining framework, which introduces (i) an iterative retraining region construction mechanism that dynamically identifies preference-violating regions in the state space via behavior bias localization; (ii) a decaying ε-scheduling strategy to jointly balance global exploration and local correction; and (iii) neural network formal verification—using ReLU partitioning and linear programming—to quantify preference adherence. Evaluated across motion control, navigation, and power grid dispatch tasks with over one hundred random seeds, our method achieves a 37.2% increase in preference compliance rate and accelerates convergence by 2.1×, significantly improving sample efficiency and policy reliability.

Offline pretraining often suffers from rapid degradation and poor exploration during early online reinforcement learning. To address this, we propose a policy expansion mechanism that treats the frozen offline policy as a fixed behavioral prior, dynamically coordinating it with a learnable online policy. Our key contribution is the first adaptive dual-policy architecture, integrating a policy-ensemble-based gating mechanism with behavioral distribution matching constraints. This ensures the offline policy remains unupdated while continuously guiding exploration, while enabling the online policy to incrementally acquire novel behaviors. Evaluated on multiple continuous-control benchmarks, our method significantly improves sample efficiency and final performance, avoids initial performance collapse, and achieves more stable convergence—outperforming standard fine-tuning and policy distillation baselines across all metrics.

This work addresses the inefficiency of existing reinforcement learning methods that often disregard available suboptimal baseline policies, resulting in high training costs and low task success rates. We propose a model-free policy augmentation framework that leverages a dynamic arbitration mechanism: during early training, control is delegated to a functional baseline policy to ensure goal reachability, and is gradually transferred to a learnable policy, ultimately yielding a high-performance policy independent of the baseline. We formally define functional baselines for the first time and integrate probabilistic reachability analysis to design the transfer mechanism, providing theoretical guarantees on the lower bound of goal achievement probability for the final policy. Experiments on continuous control benchmarks demonstrate that our method achieves competitive or superior returns compared to state-of-the-art approaches while consistently maintaining the highest goal success rate throughout both training and standalone deployment.

This work addresses the challenges of policy learning in contextual bandits with extremely large action spaces, where inefficient exploration, high variance of importance weights, and optimization difficulties commonly arise. To improve exploration efficiency in online settings, the authors propose two approaches—mixed-effects Thompson Sampling (meTS) and diffusion Thompson Sampling (dTS)—that explicitly model dependencies among actions. For offline settings, they introduce a latent-variable-based method, sDM, which integrates a differentiable pessimism mechanism with a concave policy-weighted log-likelihood objective to mitigate extrapolation bias and variance issues. Theoretical analysis yields regret bounds that scale with the effective number of actions, and empirical results demonstrate that the proposed methods significantly enhance both stability and performance of policy learning in large action spaces.

This work addresses the challenge that large language model agents struggle to effectively leverage accumulated experience during multi-step interactions, as existing approaches decouple rule-based guidance from policy learning. The paper proposes JERP, a novel framework that tightly couples experiential rule learning with policy optimization for the first time. Within JERP, both the policy model and a persistent experience rule pool are updated synchronously from the same interaction trajectory. During decision-making, the agent retrieves relevant rules and conditionally generates actions informed by historical context; post-interaction, it jointly refines the policy and revises rules using the complete trajectory. This dynamic alignment between rules and policy enables stable internalization of behavior. Empirical results demonstrate significant improvements in decision-making performance on complex tasks in the AlfWorld and WebShop benchmarks.

Existing evaluation methods struggle to assess autonomous agents’ ability to continuously refine executable policies through feedback under a limited interaction budget. To address this gap, this work proposes a novel paradigm termed “autonomous policy evolution” and introduces EvoPolicyGym—a controlled, interactive reinforcement learning benchmark that enables agents to iteratively edit and optimize their policies within a fixed budget while offering trajectory-level diagnostic capabilities to analyze budget allocation and feedback utilization mechanisms. Experimental results demonstrate that agents augmented with large language models (e.g., GPT-5.5) achieve top-two performance across all 16 environments and obtain the highest aggregate ranking, validating the framework’s effectiveness in evaluating policy evolution capabilities.

This work addresses the challenge of sparse trajectory information in long-horizon reinforcement learning for large language model (LLM) agents, where weak policies often fail repeatedly, hindering effective policy optimization. The authors propose a policy-centric training paradigm that dynamically models skills as evolving scaffolds aligned with policy development. Specifically, during inference, the framework adaptively provides guidance through evidence card generation, task-specific evaluation, and context-aware adjustment mechanisms, gradually reducing reliance on external support as the agent’s capabilities improve—thus balancing guided assistance with growing autonomy. Integrated with standard RLVR optimization, this approach outperforms strong baselines by up to 18.6% on ALFWorld and WebShop benchmarks, achieves competitive performance across seven retrieval-augmented question-answering tasks, and reduces prompt usage by 32.1%.