Adaptive Auto-Harness: Sustained Self-Improvement for Agentic System Deployment on Open-Ended Task Streams

2026-06-01

AI Summary

Existing automated optimization frameworks struggle with task heterogeneity and distribution shifts in open-ended task streams due to their reliance on a single dense update mechanism, often leading to premature performance saturation or degradation. This work proposes the first adaptive automated optimization framework that enables task-level dynamic adaptation. By decomposing the optimization gap into evolutionary loss and adaptation loss, the framework introduces a stateful multi-agent evolver and a harness tree equipped with a routing mechanism. It further integrates runtime task routing, human feedback hooks, and execution-feedback-driven continual learning. Evaluated on three open-ended task streams—prediction markets, security competitions, and event forecasting—the method significantly outperforms five strong baselines, and ablation studies confirm the effectiveness of each core component.

Abstract

Auto-harness systems such as A-Evolve, GEPA, and Meta-Harness improve LLM agents by optimizing prompts, skills, tools, memories, and supporting infrastructure from execution feedback, but they are typically evaluated on fixed offline benchmarks. Real deployments instead present open-ended task streams: histories grow without a fixed endpoint, heterogeneous tasks require different harnesses, and problem distributions shift over time. These challenges make a single repeatedly and densely updated harness brittle, causing performance degradation as accuracy peaks early and then declines. This motivates sustained harness construction with task-wise adaptation. We introduce Adaptive Auto-Harness, a framework and system for such streams. The framework decomposes the gap to an oracle harness into evolution loss and adaptation loss. The system addresses these losses with a stateful multi-agent evolver, a harness tree with solve-time routing, and human-steering hooks for cases where history lacks the needed signal. Across prediction-market, security-competition, and event-forecasting streams, Adaptive Auto-Harness outperforms five existing auto-harness baselines and ablations attribute gains to better construction, routing, or targeted human steering. Code is available in https://github.com/A-EVO-Lab/AdaptiveHarness .

Problem

• open-ended task streams
• auto-harness
• performance degradation
• heterogeneous tasks
• distribution shift

Innovation

• Adaptive Auto-Harness
• open-ended task streams
• harness tree
• solve-time routing
• human-in-the-loop steering