Flat Score, Amplified Failures: How the Error Budget Masks Damage in Quantized LLM Agents

2026-07-29

AI Summary

This work reveals that 4-bit quantization, while appearing lossless in multi-turn tool-augmented agents under standard evaluation protocols, actually exacerbates underlying error behaviors. Through extensive experiments across diverse scenarios, the authors demonstrate that conventional metrics—due to overly permissive error budgets—fail to capture genuine performance degradation. Quantization primarily amplifies pre-existing errors rather than introducing novel ones. To address this, the study proposes more sensitive evaluation criteria: tightening the error budget and measuring channel-level error rates. A comprehensive assessment framework is introduced, integrating mixed-precision quantization, equivalence testing, error-type analysis, and targeted repair prompts. Empirical results in the telecommunications domain show that 4-bit quantization can increase original errors by up to 2.5×; reducing the error budget from 10 to 2 exposes a performance gap as large as 17 points, which targeted prompts effectively mitigate.

Abstract

Post-training quantization to 4-bit weights is widely reported to be nearly lossless. We test this claim for multi-turn, tool-calling agents, where it now matters most. On $τ^2$-bench, across two open-weight model families in dense and MoE variants and two domains (eight cells, 456 episodes each, at 16-, 8-, and 4-bit weights), quantization indeed looks free on the standard metric. No cell shows a score change that survives multiple-comparison correction, and in the cell that carries the largest process damage, equivalence testing bounds the change within $\pm$7.5 points. The process tells a different story. Quantization amplifies the failure the model already exhibits at full precision (tool-name hallucination in telecom, with the same directional trend in retail entity errors) by up to 2.5$\times$ in volume (+17.6 points per task), while creating essentially no new failures. The failure set is the same at every precision (rank correlation $\geq$ 0.94, 0.18% novel events). The score stays flat because the benchmark's ten-error budget absorbs the extra failures. Shrinking the budget to two errors re-exposes a score gap of 17 points, and it does so only in the one cell where quantization added error volume, exactly as the masking account predicts. A targeted error-repair prompt, run for five telecom models at every precision, removes the damage exactly and only where it lives. Both diagnostics, the per-channel error rate and success under a shrinking budget, come from logs benchmarks already collect; we suggest reporting them alongside task reward.

Problem

• quantization
• error budget
• LLM agents
• failure amplification
• tool-calling

Innovation

• quantization
• error budget
• LLM agents
• failure amplification
• evaluation metrics