EvoHarness-RL: Learning Self-Evolving Runtime Harness for Long-Horizon LLM Agents

2026-08-05

AI Summary

This work addresses the limitations of existing long-horizon large language model agents, which rely on handcrafted prompts or heuristic rules to manage external state and struggle to effectively coordinate belief, progress, and experience. To overcome this, the authors propose a learnable harness strategy that defines an action space via supervised fine-tuning and optimizes the reading, updating, and integration of external state using a cost-aware GRPO algorithm. The approach innovatively incorporates harness annealing and evolution mechanisms, enabling the agent to transition from frequent to selective external memory access, thereby evolving the external state into a compact, task-adaptive structure. Evaluated on ALFWorld with Qwen3-8B, the method achieves a 96.9% task success rate, substantially advancing the performance of long-horizon agents.

Abstract

Long-horizon LLM agents increasingly rely on external execution support to maintain state, track progress, invoke tools, verify outcomes, and reuse experience across interactions. However, effective harness use raises two coupled challenges: state formation from noisy interaction traces and runtime control over external-state access. Existing agents usually handle both through prompts, heuristics, or domain-specific conventions, leaving the external workspace and its usage policy manually engineered. To address this, we study the problem of harness policy learning, where agents learn harness policies offline and deploy them to construct and update external harness state online during runtime task execution. We introduce EvoHarness-RL, which exposes Belief, Progress, and Experience (BPE) as policy-facing harness state. Supervised harness fine-tuning teaches the base agent the harness action space and how to construct useful external state, while cost-aware GRPO explores coordination policies to selectively read, update, and consolidate that state during long-horizon interaction. Instantiated on ALFWorld with a Qwen3-8B LLM, EvoHarness-RL reaches 96.9% success and reveals two key dynamics: harness annealing, where training internalizes recurring harness-use patterns into the model policy and shifts the agent from frequent harness calls toward selective external-state access, and harness evolution, where progress updates and experience consolidation refine the harness into a compact, task-adaptive state substrate. These results suggest that long-horizon agents benefit from trainable policies for constructing and coordinating with external harness workspaces, beyond simply adding stronger tools or larger memories.

Problem

• long-horizon agents
• external harness
• harness policy learning
• runtime state management
• LLM agent coordination

Innovation

• harness policy learning
• long-horizon LLM agents
• external state coordination
• GRPO
• self-evolving runtime harness