DFAH-Bench: Benchmarking Observable Agent Instability in Financial Decision-Making

2026-06-10

AI Summary

This work addresses a critical yet overlooked issue in financial AI agents: despite producing consistent final outputs, their tool invocations and reasoning trajectories often exhibit substantial inconsistencies that are masked when evaluation focuses solely on end results. To tackle this, the authors introduce DFAH-Bench, a novel benchmark grounded in the Determinism-Faithfulness Assurance Harness (DFAH) framework, which formally defines “faithfulness” as the consistency of observable execution across replays. The benchmark enables fine-grained assessment through two metrics—Decision Agreement Rate (DAR) and Tool-path Agreement Rate (TAR). Evaluated on synthetic compliance and financial DataOps datasets across 570 forward-looking cases, the study reveals high decision consistency (94.2–95.1%) but markedly lower consistency in tool paths (66.9–69.4%) and reasoning trajectories (45.0–51.5%), uncovering significant execution-level variability beneath stable outcomes and establishing a new paradigm for reproducible, replay-based evaluation in compliance and DataOps contexts.

Abstract

Standard evaluation benchmarks measure what a tool-using agent decides, not whether it arrives at that decision through the same process each time. We introduce DFAH-Bench, a replay benchmark that measures observable behavioral instability in financial agent decision-making across three channels -- tool-call trajectories, evidence contacts, and decision concentration -- none of which require access to hidden reasoning text. Across 8,127 replay episodes spanning 10 models and 3 financial tasks, we find that outcome agreement alone is an incomplete stability signal: frontier models can agree on decisions 95% of the time while following the same tool path only 77% of the time -- an 18-percentage-point gap (95% CI: [0.14, 0.22]) that outcome-only evaluation misses entirely. Among frontier-model case groups with high decision agreement, over 55% exhibit meaningful trajectory divergence. We identify three behavioral profiles: pattern matchers that achieve near-perfect agreement by collapsing to a single output regardless of input, stable executors with relatively consistent tool-use processes, and trajectory divergers that reach the same conclusions through materially different tool paths and evidence contacts. The benchmark code, metric scripts, replay logs, benchmark card, dataset README, and release manifest are released in the accompanying repository.

Problem

• agent instability
• financial decision-making
• observable execution
• replay consistency
• decision reproducibility

Innovation

• agent instability
• determinism-faithfulness
• replay consistency
• tool-path agreement
• financial AI benchmarking