Problem
Research questions and friction points this paper is trying to address.
harness bugs
agentic systems
bug reproduction
benchmark
software agents
AgentBug-Smith: Automatically Reproducing Real-World Harness Bugs in Agentic Systems
🤖 AI Summary
This study addresses the challenges of bug fixing in agent frameworks and the high construction costs and limited scale of existing benchmarks by proposing a large language model-driven approach for automated bug mining and reproduction. Through a novel automated pipeline, real-world bugs from open-source systems are transformed into executable instances and repair knowledge to construct Live-Harness-Bench, a scalable and dynamic evaluation benchmark. Experimental results demonstrate that this method improves the bug reproduction success rate by over 10%, establishes a benchmark comprising 200 real-world bugs, and enhances the bug-fixing performance of agents by 6.32%.
📝 Abstract
Agent harness bugs exhibit unique characteristics and remain challenging for state-of-the-art software agents to repair. Progress in this area is further hindered by existing benchmarks, which contain only a small and fixed number of executable harness bugs while requiring hundreds of human hours to construct. This work presents AgentBug-Smith, an automated harness bug reproduction approach that continuously discovers and reproduces real-world harness bugs from open-source agentic systems. Across different backbone LLMs, AgentBug-Smith consistently outperforms existing bug reproduction techniques designed for general software, achieving 10.67% - 27.56% higher success rates of reproducing harness bugs. By applying AgentBug-Smith to open-source agentic systems in the wild, we construct Live-Harness-Bench, a live and extensible benchmark that currently contains 200 reproducible harness bugs. We further demonstrate the utility of Live-Harness-Bench through two downstream applications. First, we use Live-Harness-Bench as the evaluation benchmark to systematically evaluate state-of-the-art software agents, revealing their limited capabilities in repairing real-world harness bugs. Second, we use Live-Harness-Bench as a knowledge base of real-world harness bug fixes, from which reusable repair skills can be distilled to improve existing software agents, increasing their harness-bug repair rates by 6.32%. Together, AgentBug-Smith and Live-Harness-Bench establish a scalable foundation for continuously evaluating and improving software agents on harness bug repair, turning real-world agent failures into executable evaluation instances and reusable knowledge for harness improvement, thus contributing to the ultimate goal of recursively self-improving agents.
Innovation
Methods, ideas, or system contributions that make the work stand out.
Automated Bug Reproduction
Agentic Systems
Harness Bugs
Dynamic Benchmark
Self-Improving Agents
🔎 Similar Papers
No similar papers found.
💼 Related Jobs
No related jobs found.
