Extreme Low-Bit Inference in Reasoning Models: Failure Modes and Targeted Recovery

2026-06-01

AI Summary

This work addresses the instability and repetitive generation pathologies that commonly arise in large language model inference under extreme low-bit quantization (e.g., 2-bit), which often nullify end-to-end acceleration benefits. The study is the first to attribute such failures to controllable generation pathologies and introduces two lightweight intervention mechanisms: FP16 high-precision planning and repetition-trajectory-aware selective fallback. Experiments on the Qwen3 series demonstrate that these strategies dramatically restore reasoning accuracy while preserving computational efficiency—boosting the 2-bit Qwen3-8B’s performance on MATH-500 from 17.2% to 74.2%, and achieving 87.2% with Qwen3-32B when both mechanisms are combined.

Abstract

Large Reasoning Models (LRMs) rely on long reasoning traces, making inference expensive. While low-bit quantization reduces per-token decoding cost, we show that aggressive 2-bit inference can fail to deliver end-to-end speedup because instability in the generation process inflates total token count. Instead of merely lowering answer accuracy, 2-bit quantization often produces much longer traces with repetitive loops, budget exhaustion, delayed commitment, and unclosed reasoning segments. We analyze full reasoning traces of Qwen3 reasoning models across mathematical and commonsense benchmarks and show that accuracy degradation is tightly linked to these process-level failures. To address them, we introduce two lightweight controls: FP16 planning, which gives the 2-bit model a short high-precision outline, and loop rescue, which detects repetitive traces and either commits to an earlier answer or falls back to FP16. On MATH-500, loop rescue improves Qwen3-8B accuracy from 17.2% to 74.2%, while planning plus loop rescue improves Qwen3-32B from 65.0% to 87.2%. Overall, our results show that extreme low-bit reasoning becomes practical when its failures are treated as controllable generation pathologies: with lightweight detection and selective FP16 support, 2-bit inference can recover accuracy while preserving real end-to-end speed. Our code is available at: https://github.com/brain-lab-research/quantized-reasoning.

Problem

• low-bit quantization
• reasoning models
• generation instability
• failure modes
• inference efficiency

Innovation

• extreme low-bit quantization
• reasoning trace pathology
• FP16 planning
• loop rescue
• controlled generation