When Agent Metrics Measure Different Things: An Evidence-Grounded Audit of the Praxa AI Pipeline

📅 2026-09-10
📈 Citations: 0
✨ Influential: 0
📄 PDF
🤖 AI Summary
研究通过审计Praxa AI管道文件及记录,发现代理评估指标与标签含义不一致的问题,并提供了一个可重用的验证包来区分不同类型的性能声明。
📝 Abstract
Agent evaluations can be numerically correct while measuring a different construct from the one implied by their labels. We present a retrospective measurement audit of selected Praxa AI implementation files, historical evaluation artifacts, and operational records. A 139-case offline routing report contains 112 passes and 27 failures despite zero gating failures, because known gaps are explicitly exempted from the gate. An identifier-free export represents 8,843 tool-attempt rows: 8,395 recorded durations and 448 missing values. Of the durations, 121 equal the signed 32-bit maximum and carry abandoned-client labels; inspected database code clamps elapsed lifecycle age. The pooled recorded 99th percentile is 2,147,483,647 ms, versus 38,118.31 ms among server-observed completed calls. This is a stratum contrast, not a treatment effect. In a documented single-trajectory compaction pilot, the reported follow-up input reduction is 94.39%, but the reduction across the trigger and follow-up calls together is 46.54%. We reproduce the descriptive calculations, verify 91 timing statistics through a separate weighted rational-arithmetic implementation, and execute 13 scoring-function and 12 analysis-verifier tests. Finite-completion bounds show how missing durations limit all-row timing statements without imputing values. The contribution is a source-linked case study and reusable verification package for separating gate policy, lifecycle timing, and request-level accounting from broader agent-performance claims. Historical provider runs and the full current pipeline were not independently reproduced; general capability superiority and population-level statistical significance are not established.
Problem

Research questions and friction points this paper is trying to address.

Agent Evaluations
Measurement Audit
Praxa AI Pipeline
Construct Validity
Gate Policy
Innovation

Methods, ideas, or system contributions that make the work stand out.

measurement audit
agent evaluation
reusable verification package
lifecycle timing
🔎 Similar Papers
S
Stefan G. Creadore
Tampa, FL, USA
P
Peyton Woakz
Tampa, FL, USA