Innovation
Methods, ideas, or system contributions that make the work stand out.
LLM agents
failure recovery
checkpoint evaluation
set-path symmetry
pooled success probability
Same Winners, Different Success Rates: Evaluating How LLM Agents Recover from Failures
🤖 AI Summary
This study addresses the reliability misjudgments in existing LLM agent fault-recovery benchmarks, which rely on set-consistency metrics that fail to distinguish partial successes from complete failures. Through theoretical formalization, we demonstrate the symmetry limitations of such ordinal metrics and propose a four-dimensional diagnostic framework centered on pooled success rates, validated via statistical testing and large-scale frozen-benchmark experiments. Our results reveal that conventional consistency measures may obscure deficiencies in low-success-rate scenarios, whereas the proposed framework precisely quantifies agent recovery capabilities. This work provides a robust evaluation methodology requiring no additional data.
📝 Abstract
Evaluating how LLM agents recover from mid-task failures is central to deploying reliable agentic systems. Existing checkpoint-based benchmarks measure recovery by comparing which action is selected as best across independent runs, a quantity known as set agreement. However, set agreement is a purely ordinal measure that records which action wins without reflecting the absolute level of performance. When all actions fail, they tie at zero reward, and independent runs produce the same tied set with high probability, creating an illusion of stability that masks near-zero recovery success. We formalize this limitation through a set-path symmetry result, proving that for equal-cost Bernoulli actions the success probabilities (0.9, 0.8) and (0.2, 0.1) yield identical best-action-set distributions at every sample size. No procedure based solely on which action wins can distinguish these two regimes. We further prove that certifying exact population ties is impossible in finite time, and that the assignment of outcomes to checkpoints carries information beyond marginal outcome distributions. The pooled success probability is the missing scalar that resolves the ordinal ambiguity. Experiments on 864 frozen RecoveryBench episodes and two planning cohorts totaling 3,456 responses confirm the theoretical predictions. Agreement and held-out quality can move in opposite directions, and permuting checkpoint-to-action bindings changes 8 to 13 percent of cell-level conclusions. Based on these findings, we propose reporting four diagnostic quantities (agreement, all-zero fraction, held-out success, and pooled success) that expose this failure mode with no additional data collection.
🔎 Similar Papers
No similar papers found.
