🤖 AI Summary
Current large language model evaluation benchmarks report only aggregate accuracy, offering limited insight into fine-grained capability deficiencies. This work proposes the first unified behavioral diagnostic framework that employs leave-one-model-out (LOMO) cross-validation to cluster failure modes across probe tasks in single-turn and multi-turn dialogues as well as adversarial settings, thereby constructing an interpretable taxonomy of model failures. The approach uncovers common structural patterns underlying model errors and identifies a meta-failure mode characterized by a significant gap between a model’s self-assessment and its actual performance. Experiments demonstrate that with just 50 samples across 2,664 single-turn tasks, the method achieves a Kendall’s tau of 0.81 for ranking consistency and an AUC of 0.88 for cross-model failure prediction, while yielding highly cohesive and interpretable clusters in multi-turn and adversarial scenarios.
📝 Abstract
Standard benchmarks report aggregate accuracy, but practitioners need to know which specific capabilities a model lacks. We introduce FailureScope, a behavioral-diagnosis method that clusters evaluation probes by their cross-model pass/fail patterns (leave-one-model-out, LOMO), and show it yields stable, interpretable failure taxonomies across three regimes usually studied separately: single-turn benchmarks, multi-turn dialogue, and adversarial agent attacks. On 2,664 single-turn tasks across 18 models, taxonomy-conditioned sampling reaches Kendall's tau = 0.81 at 50 tasks (versus 0.34 for random selection), and cross-model failure prediction reaches AUC 0.88. The same primitive recovers interpretable clusters on a 363-task multi-turn corpus and on 630 adversarial agent traces, where it exposes a meta-failure mode: a 73-100 percentage-point gap between LLM-judge ASR and real execution. Cluster cohesion remains strong across all three regimes, which we take as evidence that behavioral clustering is a portable diagnosis primitive that generalizes beyond any single benchmark. We release the pipeline, three annotated corpora, and the cross-regime taxonomies.