Learning Perturbation Robust Policies for LLM Agents with Stable Optimization

2026-09-28

AI Summary

This study addresses the high sensitivity of large language model (LLM) agent policies to perturbations during reinforcement learning by proposing a stable perturbation-robust policy optimization method. The approach introduces an adaptive sensitivity-aware perturbation mechanism that dynamically quantifies the policy's responsiveness to perturbations, thereby theoretically guaranteeing monotonic policy improvement and training stability. Experimental evaluations on the ALFWorld and WebShop benchmarks demonstrate that the proposed method significantly enhances the resilience of LLM agents against interference. By effectively improving policy robustness while ensuring stable convergence throughout the optimization process, this work provides a principled solution for deploying reliable LLM-based agents in complex interactive environments.

Abstract

Reinforcement learning (RL) has become an effective post-training paradigm for long-horizon large language model (LLM) agents. However, we find that the resulting policies can be sensitive to various policy perturbations, such as hidden-state noise, pruning, and quantization. In this work, we study how to improve perturbation robustness during policy optimization. We first introduce the notion of a perturbation robust policy and analyze conditions under which perturbed policy updates preserve stable monotonic improvement. Based on this analysis, we introduce Stable Perturbation-Robust Policy Optimization (SPrPO), which applies adaptive and sensitivity-aware perturbations during RL training. We evaluate SPrPO on ALFWorld and WebShop and conduct systematic experiments across multiple perturbation types and scales, showing improved perturbation robustness while maintaining stable policy optimization.

Problem

• Large Language Model Agents
• Reinforcement Learning
• Perturbation Robustness
• Policy Optimization

Innovation

• Perturbation Robustness
• Policy Optimization
• LLM Agents
• Reinforcement Learning
• Stable Optimization