🤖 AI Summary
Current large language model (LLM) protocols rely solely on end-to-end accuracy for evaluation, which fails to reveal when errors are corrected or introduced during reasoning—particularly under distribution shifts or in modular compositions where behavior becomes opaque. This work proposes a dual-outcome measurement interface that records correctness states before and after each protocol step, enabling fine-grained auditing through reusable metrics: correction rate (c) and corruption rate (γ). By integrating paired outcome measurements, difficulty-proxied conditioning, and Markov factorization tests, the method identifies three failure mechanisms—mixed shifts, presentation contamination, and state insufficiency—and supports compositional prediction and dynamic control in multi-step protocols. Experiments on synthetic mathematical tasks and GSM8K demonstrate that the interface accurately predicts optimal activation or suppression of protocol modules, substantially enhancing system predictability and robustness.
📝 Abstract
Large language models are increasingly deployed as protocols: structured multi-call procedures that spend additional computation to transform a baseline answer into a final one. These protocols are evaluated only by end-to-end accuracy, giving limited insight into when they help, when they hurt, and whether their behavior transfers under distribution shift or composition. We propose a paired-outcome measurement interface for auditing a single protocol step on exact-match tasks. For each instance, the interface records a baseline correctness bit $E_0\in\{0,1\}$ and a post-step correctness bit $E_1\in\{0,1\}$, separating correction ($E_0=0\to E_1=1$) from corruption ($E_0=1\to E_1=0$) through two rates: $c=\Pr(E_1=1\mid E_0=0)$ and $γ=\Pr(E_1=0\mid E_0=1)$. These rates predict accuracy changes and define a reusable empirical interface testable across seeds, mixtures, and pipelines. We identify three failure mechanisms. Under mixture shift, pooled estimates of $(c,γ)$ become biased when calibration and deployment mixtures differ; conditioning on a difficulty proxy restores stability without additional model calls. Under presentation contamination, selection protocols alter the interface through stable presentation artifacts when candidate content is fixed. Under state insufficiency, the correctness bit may not carry enough history for multi-step pipelines to compose predictably; a Markov factorization test identifies when composition is valid and where additional state is needed. When a protocol step passes these diagnostics, it becomes an auditable module: gated by estimated gain, conditioned on a difficulty proxy to correct mixture bias, and composed into multi-step pipelines with predictable accuracy. We demonstrate these ideas on synthetic mathematical tasks and on GSM8K, where the calibrated interface correctly predicts when protocol steps should be activated or suppressed.