One Human, $N$ Agents: Audit-Budget Allocation for LLM Agent Fleets under Miscalibrated, Correlated Confidence

2026-07-30

AI Summary

This study addresses the challenge of auditing large language model (LLM) agents under limited human auditing budgets, where self-reported confidence scores are adversarially miscalibrated and errors exhibit strong correlations. The authors formulate auditing as a budget-constrained noisy detection problem and introduce a bivariate Gaussian copula to jointly model confidence miscalibration and error correlation. Theoretical analysis reveals critical conditions under which confidence-based auditing fails: the miscalibration threshold increases as the audit budget decreases, and error correlation stems primarily from shared task difficulty rather than model lineage. Building on these insights, the work proposes a quantitative criterion for “vacuous supervision.” Experiments show that most open-source LLMs exhibit near-constant confidence exceeding the failure threshold, whereas a proprietary model demonstrates well-calibrated confidence below this threshold. Strategy replay further validates the theoretical ranking’s efficacy.

Abstract

A single human must audit $N$ LLM agents under a budget of $B \ll N$ audits per round, guided by self-reported confidence that may be adversarially miscalibrated and by correlated errors. We model this as budgeted noisy inspection over a two-level Gaussian copula and locate the miscalibration threshold $δ^*$ past which confidence-ranked auditing is \emph{worse} than random. Two a-priori expectations reverse: $δ^*$ \emph{rises} as the budget shrinks, and cross-family correlation is not low---shared difficulty dominates lineage. Five open-weight LLMs show operationally useless (near-constant) confidence, point estimates at or beyond the flip though CIs straddle it; a proprietary model is informative and lands below it. We give a quantitative criterion for \emph{vacuous} oversight, and replaying policies on recorded traces confirms the ordering.

Problem

• audit-budget allocation
• LLM agent fleets
• miscalibrated confidence
• correlated errors
• human oversight

Innovation

• miscalibrated confidence
• correlated errors
• budgeted auditing
• Gaussian copula
• LLM agent fleets