SearchAuditor: Auditing and Attributing Failures in Long-Horizon Search Agents

2026-08-05

AI Summary

This work addresses the challenge that long-horizon search agents in complex network environments are prone to catastrophic failure due to minor reasoning errors, while manual diagnosis of lengthy execution trajectories remains prohibitively costly. To tackle this, the authors introduce SearchAuditBench—a benchmark comprising 1,243 systematically annotated failure trajectories with root causes and corresponding repair strategies—and propose SearchAuditor, a multi-perspective auditing framework. SearchAuditor integrates large language model–driven multi-view analysis, evidence-guided error adjudication, and structured repair generation to substantially reduce human intervention. Experiments demonstrate that SearchAuditor achieves a 32.3% end-to-end success rate across multiple state-of-the-art models, significantly outperforming the strongest baseline (26.6%) and effectively enabling agent recovery and resumption.

Abstract

Deep search agents tackle challenging questions through long-horizon web interactions, a process that is both complex and fragile: small reasoning errors may propagate through long, noisy trajectories into fluent but incorrect answers. Diagnosing such failures is difficult, requiring the manual inspection of extremely long execution traces, which could be beyond human capacity. We therefore introduce SearchAuditBench, a benchmark that evaluates whether LLM auditors can localize, attribute, and repair these failures, thereby reducing the human burden. SearchAuditBench comprises 1,243 failed trajectories, averaging 73.1 messages and 65.1K tokens, collected from eight open-weight models on five deep-search benchmarks, each expert-annotated with the critical error step, a search-specific root cause, and a reference repair with grading rubrics. We further propose SearchAuditor, a multi-perspective auditing framework that effectively localizes, attributes, and repairs search-agent failures through evidence-grounded adjudication. Experimental results show that even the strongest baseline, when powered by a frontier model like GPT-5.5, attains only a 26.6% end-to-end pass rate. In contrast, our SearchAuditor consistently outperforms all baselines across different frontier models, achieving an end-to-end pass rate of 32.3%, and resuming failed runs with its repairs enables agents to better recover from errors.

Problem

• long-horizon search agents
• failure attribution
• execution trace auditing
• error diagnosis
• LLM auditors

Innovation

• SearchAuditor
• long-horizon search agents
• failure attribution
• evidence-grounded adjudication
• SearchAuditBench