trust-region optimization

Standard PPO struggles to transfer policies effectively in non-stationary environments due to its geometrically unaware local updates and excessive regularization that overly suppresses necessary policy adjustments. This work proposes Gaussian Trust Region (GTR) optimization, which dynamically reshapes the trust region using a Gaussian kernel to enable adaptive, large-magnitude updates along high-advantage directions while preserving local stability. Additionally, GTR introduces dynamically blended Gaussian anchor policies to mitigate variance caused by outdated reference policies. The resulting framework is architecture-agnostic and demonstrates consistent and significant improvements over strong baselines across diverse domains—including gaming, robotic control, open-world exploration, and language model post-training—highlighting its generality and robustness in non-stationary settings.

This work addresses the issue that standard entropy regularization in reinforcement learning for large language models often leads to ineffective exploration or even performance degradation due to the accumulation of tail risks. To mitigate this, the authors propose Trust Region Entropy (TRE), a novel approach that, for the first time, integrates trust region constraints into entropy regularization. By maximizing entropy within a local neighborhood of the current policy, TRE focuses exploration on high-confidence candidate tokens, thereby enhancing both exploration efficiency and reasoning coherence. Implemented within the PPO framework, the method combines localized entropy maximization with trust region constraints and demonstrates significant improvements over standard PPO, conventional entropy regularization, and other exploration baselines across benchmarks including MATH, Countdown, and HH.

In reinforcement learning for large language models, policy gradient methods suffer from approximation errors that scale quadratically with sequence length $T$ ($O(T^2)$) due to policy mismatch, and conventional trust-region methods fail on long-horizon tasks. Method: We derive the first tight trust-region error bounds of $O(T)$ and $O(T^{3/2})$, and propose Trajectory-level Regularization Masking (TRM), a trajectory-wise masking mechanism that dynamically discards entire sampled trajectories based on the maximum token-level KL divergence within each trajectory, monitored via full-sequence KL divergence. Contribution/Results: TRM establishes the first sequence-level policy optimization framework with nontrivial monotonic improvement guarantees. Experiments demonstrate significantly enhanced policy stability and convergence on long-horizon tasks, overcoming the trust-region breakdown inherent in token-independent methods like PPO, while providing verifiable theoretical guarantees.

This work addresses the instability in existing GRPO-based reinforcement learning fine-tuning methods for large language models, which rely on heuristic trust region approximations and struggle to effectively constrain importance ratios that exceed clipping bounds. To overcome this limitation, the authors propose QUATRO, a principled optimization approach that enforces explicit trust region constraints and introduces a query-adaptive mechanism. This mechanism derives intrinsic stabilization terms directly from the exact trust region formulation, enabling controlled policy updates and stable entropy regulation. Experimental results across multiple mathematical reasoning benchmarks demonstrate that QUATRO maintains training stability even under high policy staleness and large step sizes, effectively controls policy entropy, and significantly enhances both optimization robustness and task performance.

To address the reasoning performance bottleneck in LLMs arising from imitation rigidity, suppressed exploration, and capability forgetting during two-stage training (SFT → RL), this paper proposes a dynamic interleaved unified optimization framework. Methodologically, it integrates supervised fine-tuning (SFT), reinforcement learning (RL), trust-region optimization, and KL-divergence control. Its core innovations are: (1) Trust-Region Supervised Fine-Tuning (TrSFT), which ensures stable parameter updates under forward KL constraints and adaptively switches to reverse KL for out-of-distribution samples; and (2) a utility-driven dynamic expert prefix selection mechanism that schedules SFT or RL optimization paths token-wise based on input prefixes. Evaluated on five mathematical reasoning benchmarks, the method consistently outperforms standard SFT, RL, SFT-then-RL, and current state-of-the-art approaches, delivering significant gains in complex reasoning capabilities.

This work addresses the challenges of high-dimensional black-box constrained optimization, where function evaluations are expensive, gradient information is unavailable, and the feasible region is complex. The authors propose a Bayesian optimization method that integrates a penalty function with a trust-region mechanism. By incorporating constraints into an unconstrained formulation via penalty terms, the approach constructs a local Gaussian process surrogate model and performs sampling within a dynamically adjusted trust region using the expected improvement criterion, thereby effectively balancing exploration and exploitation. Experimental results demonstrate that the proposed method consistently achieves high-quality feasible solutions with significantly fewer function evaluations than state-of-the-art approaches across multiple high-dimensional synthetic and real-world constrained optimization problems, yielding notable improvements in both sample efficiency and optimization stability.

Existing inference-time alignment methods are often constrained by differentiability requirements, dependence on specific model architectures, or poor computational efficiency. This work proposes Trust-Region Search (TRS), a noise-based optimization algorithm that treats both the generative model and the reward model as black boxes, optimizing only the initial noise vector to strike an effective balance between global exploration and local exploitation. TRS requires no access to internal model structures, is compatible with diverse generative frameworks—including diffusion models and flow models—and supports arbitrary reward functions, all while requiring minimal hyperparameter tuning. Empirical results across text-to-image generation, molecular design, and protein engineering demonstrate that TRS consistently outperforms existing baselines and sophisticated trajectory optimization approaches, highlighting its remarkable versatility and efficiency.

This work addresses the high computational cost of traditional inverse reinforcement learning, which requires repeatedly solving full reinforcement learning problems to ensure monotonic performance improvement, and the instability and lack of monotonicity in adversarial approaches. The authors propose a trust-region-based explicit dual optimization framework that jointly optimizes the reward function and policy within a local neighborhood of the current policy, thereby avoiding complete RL solves. A key theoretical insight is that small-step updates along the reward gradient direction preserve global optimality through locally optimal policies, achieving a balance among monotonicity, stability, and computational efficiency while recovering generalizable canonical reward functions. Experiments demonstrate that the method outperforms state-of-the-art imitation learning approaches by 2.4× in aggregate performance across multiple complex tasks, and the learned rewards generalize effectively to environments with altered dynamics.

This work addresses the challenge that Gaussian process-based Bayesian optimization often converges to local optima and exhibits poor global search efficiency when applied to multimodal or high-dimensional black-box functions. To overcome this limitation, the authors propose a trajectory-aware trust-region Bayesian optimization method that models performance prediction across multiple local optimization trajectories as a best-arm identification (BAI) problem. By framing region selection within a multi-armed bandit framework, the algorithm dynamically eliminates suboptimal regions and concentrates exploration on promising areas. The proposed approach offers theoretical guarantees of faster convergence compared to conventional Bayesian optimization and demonstrates significantly improved global optimization performance on both synthetic and real-world benchmark tasks.

This work addresses the challenge of stale samples in asynchronous reinforcement learning, where policy lag, engine latency, and expert routing heterogeneity lead to high policy mismatch that undermines existing trust region methods. The paper proposes Staleness-Adaptive Trust Region (SAT), which uniquely integrates trust region mechanisms with staleness awareness by employing a detached log-probability ratio as a staleness proxy to identify high-mismatch tail samples within a batch. SAT selectively contracts only the sign-chosen endpoint of the PPO clipping interval, imposing conservative constraints on highly stale samples while preserving standard updates for others. Combined with kernel scaling and sign-based selection, SAT achieves local interval containment and pointwise pessimism, reshaping update geometry under heterogeneous staleness. Evaluated on Qwen3-30B with SGLang and Megatron, SAT-GSPO w/ R3 attains state-of-the-art performance on AIME24 avg@8 (35.83 at lag 1, 34.79 at lag 8), significantly enhancing asynchronous training stability.