Learning Meta-Skills for Agent Harness Design in Test-Time AI4AI

2026-09-29

AI Summary

This study addresses how to enhance agent performance by constructing superior execution environments while keeping model weights frozen. To this end, this work proposes a test-time AI4AI framework grounded in the Meta-Skill principle, wherein a Builder learns from feedback to construct a Harness that transforms empirical insights into reusable execution support mechanisms, thereby achieving system-level self-optimization during inference. Furthermore, the authors introduce Harness-Bench, an evaluation benchmark designed to validate the proposed approach. Experimental results demonstrate that the framework yields an 8.95 percentage point improvement in macro-average performance, effectively substantiating the potential for AI self-improvement through environment reconstruction under frozen-weight constraints.

Abstract

Agent performance depends on both reasoning ability and the environment in which it acts. We study test-time AI-for-AI, asking how a Builder can learn to construct better execution environments for a Target while both models' weights remain fixed. To make the Builder's experience reusable, we introduce Meta-Skill: principles specifying when support is needed and what resources to provide. The Builder learns these principles from Target's execution feedback on the development set, then uses the frozen skill bank to construct harnesses for unseen tasks. Across Harness-Bench and NewtonBench, full-bank meta-skills improve macro-average performance by 8.95 percentage points over no-skill construction, and 12.02 points over direct delivery of the same bank to the Target. These results highlight the value of translating experience into executable support. Gains when the same model serves both roles further suggest a path to system level self-improvement through learning to build better environments.

Problem

• Test-Time AI4AI
• Agent Harness Design
• Meta-Skills
• Execution Environment
• Self-Improvement

Innovation

• Meta-Skill
• Test-Time AI4AI
• Agent Harness Design
• Execution Environment
• Self-Improvement