proximal policy optimization

This work addresses the trade-off between training stability and sample efficiency in deep reinforcement learning, where Proximal Policy Optimization (PPO) exhibits stable on-policy learning but suffers from low sample efficiency, while off-policy methods, though more efficient, are prone to estimation bias and variance. To reconcile these limitations, the paper proposes ExO-PPO, which constructs an expectation-based lower bound for generalized policy improvement, introduces a piecewise exponential clipping mechanism, and incorporates a multi-policy replay buffer to reuse historical trajectories. This approach preserves the stability of on-policy optimization while substantially enhancing sample efficiency. Experimental results demonstrate that ExO-PPO consistently outperforms standard PPO and its state-of-the-art variants across a range of tasks, achieving significant improvements in stability, sample efficiency, and overall performance.

This work addresses the long-standing lack of theoretical unification between the two predominant variants of Proximal Policy Optimization (PPO)—namely, PPO with a clipped surrogate objective (PPO-Clip) and PPO with KL divergence penalty. By analyzing the per-sample KL divergence, the study reveals that PPO-Clip implicitly implements a sample-wise KL regularization with a stepwise coefficient. Through closed-form derivations involving the importance sampling ratio and the advantage function, the authors construct an adaptive KL penalty coefficient that exactly reproduces the gradient updates of PPO-Clip. Empirical validation on five MuJoCo continuous control tasks demonstrates nearly identical training curves between this reformulation and the original PPO-Clip, confirming their theoretical equivalence. This insight not only provides a unified interpretation of PPO but also opens new avenues for algorithm design based on adaptive KL regularization.

This work addresses the long-standing lack of rigorous convergence theory for Proximal Policy Optimization (PPO) and clarifies the theoretical underpinnings of its multi-epoch minibatch update mechanism. The authors interpret PPO updates as an approximate policy gradient ascent procedure with controlled bias and, by incorporating stochastic reshuffling techniques, establish the first convergence proof framework for PPO under standard assumptions. Furthermore, they identify a weight collapse issue in truncated Generalized Advantage Estimation (GAE) at episode boundaries and propose a corrective modification. Both theoretical analysis and empirical evaluation demonstrate that the proposed correction significantly enhances PPO’s performance in environments with strong terminal signals, such as Lunar Lander.

This work addresses the high computational cost and numerical instability encountered when extending policy-based reinforcement learning algorithms like PPO to continuous normalizing flow (CNF) policies, which typically require likelihood evaluation along full flow trajectories. To overcome these challenges, the authors propose PolicyFlow, a novel on-policy algorithm that approximates importance ratios by leveraging velocity field variations along a simple interpolation path, thereby circumventing the need for full trajectory likelihood computation. Additionally, PolicyFlow incorporates a lightweight Brownian regularization term, inspired by Brownian motion, to implicitly enhance policy diversity and mitigate mode collapse. Experimental results demonstrate that PolicyFlow matches or surpasses the performance of Gaussian PPO and flow-based baselines such as FPO and DPPO across diverse environments—including MultiGoal, PointMaze, IsaacLab, and MuJoCo Playground—with particularly strong capabilities in modeling multimodal action distributions.

Deep neural networks struggle to directly optimize non-differentiable objectives (e.g., IoU, BLEU, reward signals) due to the absence of well-defined gradients. Method: This paper reframes deep reinforcement learning (DRL) as a generalized extension of supervised learning, centering on gradient-based policy optimization frameworks (e.g., PPO) rather than tabular RL paradigms. It systematically integrates loss proxy modeling, policy gradient derivation, and human feedback alignment techniques to bridge the gap from single-step non-differentiable optimization to multi-step sequential decision-making. Contribution/Results: The work establishes, for the first time, a conceptual continuity between supervised learning and DRL—clarifying theoretical foundations, algorithmic boundaries, and practical implementation logic. This significantly lowers the entry barrier to DRL, enabling researchers with only supervised learning background to rigorously understand, adapt, and deploy state-of-the-art DRL methods across diverse application domains.

To address the low sample efficiency and high training cost of reinforcement learning in physics simulation environments, this paper proposes PPOPT: a pretraining-based, model-agnostic Proximal Policy Optimization (PPO) algorithm. Its core innovation is a segmented neural network architecture wherein intermediate layers are jointly pretrained across multiple similar physical environments to acquire transferable dynamics representations; downstream tasks then require only fine-tuning of input and output layers, drastically reducing interaction samples needed in the target environment. Experiments demonstrate that PPOPT achieves faster convergence and more stable policies under extremely limited samples (<10⁵ interaction steps), outperforming standard PPO in final performance while incurring significantly lower computational overhead than model-based methods. The implementation is publicly available.

This work addresses the disconnect between trust region theory and the heuristic clipping objective in Proximal Policy Optimization (PPO) by introducing the Bounded Regularized Reinforcement Learning (BRRL) framework. The authors formulate a constrained regularized policy optimization problem, derive its closed-form optimal solution, and design the Bounded Policy Optimization (BPO) algorithm to minimize the advantage-weighted divergence between the parameterized policy and this solution. They further extend BPO to Generalized BPO (GBPO) for large language model (LLM) fine-tuning. This study provides the first theoretical justification for PPO-style methods, unifying trust region optimization with a cross-entropy perspective while guaranteeing monotonic policy improvement. Experiments demonstrate that BPO and GBPO consistently match or outperform PPO and GRPO across MuJoCo, Atari, IsaacLab, and LLM fine-tuning benchmarks in both stability and final performance.

To address the poor training stability and slow convergence of Adaptive Neuro-Fuzzy Inference System (ANFIS) policies in reinforcement learning, this paper proposes an on-policy actor-critic framework based on Proximal Policy Optimization (PPO) for end-to-end training of ANFIS controllers. Departing from the off-policy paradigm of Deep Q-Networks (DQN), the approach preserves the interpretability of fuzzy rules while significantly enhancing training robustness and convergence efficiency. Extensive experiments across multiple random seeds in the CartPole-v1 environment demonstrate that the proposed ANFIS-PPO agent consistently achieves the maximum return of 500 after 20,000 policy updates (zero variance), outperforming the ANFIS-DQN baseline. The key contribution lies in the first integration of PPO’s gradient clipping and trust-region optimization mechanisms into joint parameter learning of ANFIS, thereby unifying interpretability with high control performance.

Proximal Policy Optimization (PPO) frequently suffers from directional bias in importance ratios—i.e., ratios decrease under positive advantages and increase under negative advantages—undermining optimization stability and performance, yet this phenomenon has lacked systematic investigation. Method: We propose Directional-Clamp PPO, the first method to explicitly model and suppress such directional deviation. It introduces a direction-aware clipping mechanism featuring a tunable threshold β and asymmetric, steep loss gradients that penalize only updates moving in the “wrong” direction, while preserving ratios near unity. The method seamlessly integrates into the standard PPO framework and remains compatible with importance sampling and advantage estimation. Contribution/Results: On multi-task MuJoCo benchmarks, Directional-Clamp PPO significantly outperforms vanilla PPO and leading variants, demonstrating consistent robustness across random seeds. Theoretical analysis confirms its ability to provably avoid harmful policy updates, thereby enhancing the reliability of policy optimization.