The Undetected Damage of Quantization on Retrieval and How to Fix It

2026-09-21

AI Summary

论文探讨了量化对检索结果的未被察觉的影响，并提出通过测量最高分与次高分之间的差距来判断量化答案的可信度，以及如何在不同任务中针对性地解决问题。

Abstract

We show that a quantized model that keeps its classification accuracy still changes $14$ to $46\%$ of its top-1 retrieval results, and that aggregate ranking metrics reveal only part of this damage. We tie this failure to the gap between the two highest scores and use that gap to decide when a quantized answer can be trusted and where additional precision should be spent. We show that the top-1 result is guaranteed to survive quantization only when this gap exceeds twice the largest rounding error. In classification, scores are the logits, and the loss function pushes the correct class away from other classes, encouraging this gap. In retrieval, scores are query-document scores, and nothing separates the top-1 item from the second. This gap can be measured without labels. Before deployment, it predicts which models will break under quantization, and at deployment time it tells, per input, whether the quantized answer still matches the full-precision answer. Most classification inputs have a gap wide enough to trust the quantized answer, but few retrieval queries do. That gap motivates a different fix in each task. In retrieval, spending extra bit-width on the layers whose quantization moves the gap most recovers up to three-quarters of an extra bit's benefit for half its cost. In classification, routing the few low-gap inputs to full precision recovers most of the lost accuracy at a fraction of the cost.

Problem

• quantization
• retrieval
• classification accuracy
• ranking metrics
• gap

Innovation

• quantization
• retrieval accuracy
• gap between top scores
• bit-width allocation
• full-precision routing