pipeline evaluation

Current complex LLM agent benchmarks are often prone to misjudgment due to specification errors, implicit assumptions, or rigid evaluation scripts, mistakenly attributing benchmark flaws to agent failures. This work proposes the first automated auditing framework leveraging state-of-the-art large language models to cross-validate task-oriented, execution-driven benchmark components through structured LLM protocols, augmented by agent solutions and execution traces for diagnostic support. The approach establishes a novel AI-assisted paradigm for benchmark validation, overcoming the limitations of traditional manual review. Applied to ScienceAgentBench, it identified 12 author-confirmed issues—including critical errors—and reproduced 83.3% of expert-discovered problems on the BIXBench Verified-50 subset, with an auditing cost of under $15 for 50 bioinformatics tasks.

Existing approaches struggle to effectively quantify the similarity and quality between synthetic and real data in evaluating tool-augmented agents. To address this gap, this work proposes SynAE, a novel framework that establishes the first multi-axis evaluation system tailored for multi-turn tool-use scenarios. SynAE introduces four fine-grained metric categories—assessing task instructions, tool invocations, final outputs, and downstream evaluation performance—to systematically measure synthetic data across dimensions of validity, fidelity, and diversity. Integrating natural language processing, trajectory modeling, and controllable generation techniques, the framework enables a reproducible evaluation pipeline and successfully identifies several representative failure modes in synthetic data generation. Empirical results demonstrate that such multidimensional assessment is essential for enhancing the reliability of agent evaluations.

Traditional benchmarks provide only aggregate scores, offering insufficient evidence to support reliable deployment decisions and thereby creating a disconnect between evaluation and action. To address this gap, this work proposes a “deployment-completeness” benchmarking framework, introducing novel metrics—evidence fibers, completeness curves, and certifiable proportions—alongside a systematic audit methodology comprising evidence fiber analysis, response ranking intervals, conformal coverage evaluation, and a certify-then-acquire decision pipeline. Empirical evaluation on benchmarks such as Tox21, Matbench, and JARVIS reveals that conventional approaches suffer a drastic drop in channel coverage to 10.07% under real-world deployment conditions. In contrast, the proposed method reduces error-driven deployment decisions to 0.027% on Tox21 and 0.128% on JARVIS, substantially enhancing deployment reliability.

This study addresses a critical gap in existing tool-calling evaluation benchmarks: the lack of validation of the evaluators themselves, which risks conflating assessment artifacts with agents’ true capabilities. Through a systematic audit of four prominent benchmarks—BFCL v4, τ2-Bench, LiveMCPBench, and MCP-Atlas—the authors conduct expert review of 496 tasks, replicate experiments, and perform trajectory-level analysis, revealing an 18.5% disagreement rate between automated evaluators and human judgment. Notably, LiveMCPBench exhibits a score variance of up to 18.9 percentage points upon re-evaluation, sufficient to overturn leaderboard rankings. To address these issues, the work introduces the first unified taxonomy of tool-calling evaluation failures, advocates for distinct measurement of tool invocation, task completion, and result verification, and releases Tool-Veritas—a configurable benchmark—and Harness Lab, an open-source evaluation platform.

This study addresses the underexplored engineering challenges in existing machine learning evaluation frameworks, where operational issues and their root causes have lacked systematic investigation. To bridge this gap, the work formally establishes evaluation engineering as a distinct research direction within software engineering. Through an empirical analysis of 57 frameworks and a comprehensive categorization of 16,560 reported issues across a newly proposed five-stage workflow model, the study reveals that 41.4% of problems originate in the specification phase, while 61.7% of classified issues stem from missing functionality, inadequate documentation, and insufficient input validation. The findings yield a structured taxonomy of evaluation-related problems and provide empirical evidence to inform the design and improvement of robust evaluation systems.

Existing benchmarks struggle to evaluate the impact of harness components on the performance of large language model (LLM) agent systems, often overlooking execution details or fixing harness configurations. This work proposes Harness-Bench, the first framework to systematically decouple and quantify how harness design influences agent workflows. By enforcing a unified task environment, computational budget, and evaluation protocol—and leveraging sandboxed offline tasks, realistic usage patterns, human auditing, and full execution trace logging—it enables controlled experiments across diverse model–harness combinations. Analysis of 5,194 execution traces across 106 tasks reveals significant differences in completion rates, process quality, efficiency, and failure modes, exposing alignment failures stemming from misalignment between reasoning and execution. The study argues that agent performance should be reported based on joint model–harness configurations rather than base models alone.