NLPG: Natural-Language Policy Gradients for Self-Evolving Language Agents

2026-09-27

AI Summary

This study addresses the performance bottlenecks of fixed-parameter language agents caused by localized decision errors and the limited reusability of their procedural experience. To overcome these limitations, this work proposes an external strategy memory approach that introduces a novel natural language policy gradient mechanism requiring no modifications to model parameters or program structures. By leveraging execution trajectory diagnosis, module graph feedback propagation, and strategy aggregation, the method transforms failures into localized natural-language corrections, enabling continuous evolution and interpretable experience reuse for frozen agents. Evaluated across six benchmarks, the proposed approach outperforms the strongest baseline by an average of 8.71 percentage points, demonstrating its effectiveness in enhancing the capabilities of fixed-parameter language agents.

Abstract

Large language model agents increasingly rely on compound programs for retrieval, tool use, reasoning, and verification, yet their failures often arise from local procedural decisions. Existing reinforcement-learning and prompt-optimization approaches typically rely on scalar rewards or repeatedly modify entire prompts, making it difficult to capture and reuse procedural improvements while preserving a frozen agent. To address this problem, We propose Natural-Language Policy Gradients (NLPG), an external policy-memory method for improving a fixed agent without changing its model parameters or program structure. NLPG diagnoses execution traces, propagates downstream feedback backward through the module graph, and converts recurring failures into route-local natural-language corrections that are aggregated into bounded policy updates for subsequent executions. Across six benchmarks covering memory, reasoning, instruction following, and evidence verification, NLPG also outperforms the strongest listed baseline for each benchmark by 8.71 percentage points on average. These results provide evidence that evaluated procedural experience can be transformed into local and interpretable policy updates, enabling continual improvement of frozen agents.

Problem

• Large Language Model Agents
• Procedural Failures
• Frozen Agent
• Reinforcement Learning
• Prompt Optimization

Innovation

• Natural-Language Policy Gradients
• Self-Evolving Language Agents
• External Policy Memory
• Execution Trace Diagnosis
• Frozen Agent Improvement