Quantized Reasoning Models Think They Need to Think Longer, but They Do Not

2026-05-29

AI Summary

This work identifies and addresses a previously unreported phenomenon in post-training quantization: “overthinking” errors, where quantized models arrive at correct intermediate answers but produce incorrect final outputs due to redundant reasoning steps. The study reveals that quantized models disproportionately generate uncertainty-associated filler tokens—such as “wait” or “but”—at high-entropy positions, leading to unnecessarily long chains of thought. To mitigate this, the authors propose a training-free logit penalty mechanism that selectively suppresses such redundant tokens. Evaluated across five models (1.5B–32B), three quantization schemes, and five reasoning benchmarks, the method reduces reasoning length by 12–23%, decreases overthinking errors by up to 58%, and maintains or improves overall accuracy, achieving a Pareto-optimal trade-off between efficiency and performance.

Abstract

Post-training quantization (PTQ) is widely used to deploy large language models efficiently, but its effect on reasoning models is not well understood. Across math, coding, and science QA, we find that aggressive PTQ reduces accuracy while increasing chain-of-thought (CoT) length. Surprisingly, we show that in up to 52% of the quantized models' failures, models reach the right answer in intermediate reasoning steps but do not output it as a final answer. To understand why quantization leads to this increase in overthinking errors, we measure the token-level KL divergence between quantized and full-precision output distributions. Positions with high KL divergence correlate strongly with high next-token entropy, and at these positions quantized models disproportionately sample overthinking markers such as "wait", "but", and "alternatively". We show that simply introducing a training-free logit penalty on a curated set of overthinking markers can reduce CoT length by 12--23% while preserving or improving accuracy across 5 models (1.5B-32B parameters), 3 quantization methods, and 5 benchmarks, yielding a favorable Pareto frontier of accuracy against reasoning cost compared to penalizing other token sets. Overthinking errors produced by quantized models are particularly reduced by up to 58%.

Problem

• quantization
• reasoning models
• overthinking
• chain-of-thought
• post-training quantization

Innovation

• post-training quantization
• chain-of-thought
• overthinking errors
• logit penalty
• KL divergence