Local Guidance, Global Impact: Gaussian-Reshaped Trust Region Unlocks Behavior Transitions

2026-06-02

AI Summary

Standard PPO struggles to transfer policies effectively in non-stationary environments due to its geometrically unaware local updates and excessive regularization that overly suppresses necessary policy adjustments. This work proposes Gaussian Trust Region (GTR) optimization, which dynamically reshapes the trust region using a Gaussian kernel to enable adaptive, large-magnitude updates along high-advantage directions while preserving local stability. Additionally, GTR introduces dynamically blended Gaussian anchor policies to mitigate variance caused by outdated reference policies. The resulting framework is architecture-agnostic and demonstrates consistent and significant improvements over strong baselines across diverse domains—including gaming, robotic control, open-world exploration, and language model post-training—highlighting its generality and robustness in non-stationary settings.

Abstract

While Proximal Policy Optimization (PPO) demonstrates strong performance in stationary settings, we show that its standard optimization paradigm struggles in continual and non-stationary environments. The failure does not stem from insufficient model capacity or overly restrictive clipping. Instead, PPO performs persistent, directionally inefficient local updates, which indicates a lack of geometry-aware guidance for accumulating meaningful behavioral change and ultimately hindering transitions toward new behavior patterns. Although divergence-based regularization introduces partial geometric awareness, its monotonically increasing penalties implicitly discourage large policy deviations, even when such shifts are necessary for effective adaptation. To address this limitation, we propose Gaussian Trust Region Policy Optimization (GTR), which reshapes the trust region using a Gaussian kernel. The resulting constraint is bounded and non-monotonic, providing strong local stability while progressively relaxing under sustained high-advantage updates. To further improve robustness, we introduce a Mixture Gaussian Anchor that adapts to recent policy trajectories, reducing variance induced by stale references. GTR is architecture-agnostic and achieves strong performance across games, simulated robotic control, open-world exploration, and language model post-training. These results demonstrate that geometry-aware trust-region design can be a promising direction for robust reinforcement learning in complex non-stationary environments. Our code is available at https://anonymous.4open.science/r/GTR_demo/README.md.

Problem

• non-stationary environments
• behavior transitions
• trust region
• policy optimization
• geometric awareness

Innovation

• Gaussian Trust Region
• non-stationary reinforcement learning
• geometry-aware optimization
• policy adaptation
• trust region reshaping