🤖 AI Summary
This study addresses a critical flaw in existing replay-based log stitching methods for evaluating multi-step LLM agents: the erroneous assumption that switching models does not alter subsequent trajectories, leading to distorted evaluations. The authors conduct controlled branch-rollback experiments in the real-world SWE-bench environment to reconstruct agent trajectories and systematically assess the actual impact of model substitutions on downstream behavior. Their findings reveal that model switching rewrites 61–94% of subsequent actions, causing replay-based evaluation to mispredict all key outcomes—trajectory similarity drops to merely 0.00–0.11. Furthermore, quantization schemes such as FP8 and AWQ significantly affect apparent “determinism.” These results expose severe bias in static replay evaluation and advocate for a shift toward dynamic evaluation paradigms. The authors publicly release all trajectories and their evaluation framework.
📝 Abstract
LLM routers promise efficiency by matching each request to the cheapest adequate model, and are increasingly applied per step inside multi-step agents. Yet agentic routers are evaluated like single-turn routers: by replaying logged trajectories and substituting another model's recorded outputs, assuming the rest of the trajectory is unaffected. We test this assumption with branching rollouts: we fork live SWE-bench agent trajectories at controlled points, rebuild the environment, continue each fork with a different model, and compare against same-model control forks that isolate sampling and replay noise. Across six paired runs (~900 rollouts), swaps exceed their matched control floors by +0.25 to +0.66 normalized edit distance (multiplicity-corrected CIs exclude zero), rewriting 61-94% of post-fork actions; 74-77% of early swaps diverge at the first post-fork action, versus 6-35% of controls, leaving only 3% of replayed states valid. Divergence decreases with fork depth in both directions. All five outcome flips we observe occur in swap arms, upgrades rescuing unsolved instances and a downgrade losing the sole solve, and zero occur across 359 control forks. Scoring these same swaps with a log-stitching replay evaluator, replay mispredicts every success-relevant outcome call and predicts patches with 0.00-0.11 similarity to reality. Auditing the noise floor, temperature-0 "determinism" is configuration-dependent: FP8-served controls diverge on over 90% of forks while AWQ-served ones remain near-identical; and under tight budgets the stronger model more often exhausts its steps without submitting. Replay-based benchmarks score the wrong world for agentic routing; we release our harness and all trajectories.