Innovation-Residual Auditing of Autonomous Analysis Agents: Localization, Detection Limits, Error Control, and Identifiability

2026-08-05

AI Summary

This work addresses the challenge of error localization in autonomous analytical agents, which often lack unsupervised auditing mechanisms when failures occur during end-to-end data analysis. The authors propose a novel anomaly detection method that requires no error annotations by modeling normal analytical behavior and assigning anomaly scores to deviant operations. Key contributions include elucidating the relationship between error localization and scoring strategies, introducing a reconstruction-length-based approach to quantify error propagation, establishing a false positive control mechanism relying solely on exchangeability assumptions, and deriving the first theoretical lower bound on error identifiability. Both theoretical analysis and experiments demonstrate that single errors can be flagged either locally or through diffusion, false positive rates remain controllable, and identification capability is primarily constrained by representation dimensionality rather than training data volume.

Abstract

Autonomous agents now carry out entire data analyses, selecting cohorts, joining tables, and fitting models with little step-by-step supervision. When such an analysis turns out to be wrong, someone must determine which operation caused it. A recent approach does this without any labelled mistakes, learning instead from analyses known to be sound and flagging operations that depart from what that model predicts; how reliable such audits are has not been studied. This paper supplies that analysis. The choice of score determines whether an error can be localized at all. If each operation is scored by how surprising it is given the operation immediately preceding it, then operations that merely inherit an earlier error are indistinguishable from correct ones, so one mistake produces one flag; scores computed against a longer reconstruction of the intended analysis instead spread a single mistake across many operations. We quantify how far they spread, and how to choose the comparison length when an error accumulates gradually rather than at once. We then give procedures that control the proportion of falsely flagged operations within a single audited analysis, requiring only that sound analyses be exchangeable rather than that the fitted model be correct, and we quantify how much the guarantees weaken when the model is imperfect or when the analysis was selected for review in a way that depends on its content. Finally we establish a limit on what any such audit can report: errors below a certain magnitude cannot be attributed at all, being indistinguishable from ordinary variation among sound analyses. This limit falls so slowly as more sound analyses are collected that at the representation sizes now in use a hundredfold increase reduces it by under two percent, so the dimension of the representation rather than the volume of training data is the binding constraint.

Problem

• autonomous analysis agents
• error localization
• innovation-residual auditing
• identifiability
• false discovery control

Innovation

• residual auditing
• error localization
• identifiability limit
• exchangeability
• autonomous analysis agents