QuantiBias: Benchmarking Quantization-Induced Bias in LLMs

2026-07-23

AI Summary

Although quantization-based compression is widely adopted for large language models, it can significantly exacerbate stereotypical biases in open-ended generation while still passing standard safety evaluations. This work proposes QuantiBias, a benchmark that constructs multilingual generative probes combining refusal control and multiple-choice tasks to systematically assess the impact of quantization on bias in open-ended outputs. Experiments across models such as Qwen and Gemma, and various quantization schemes, reveal that 24%–27% of quantized model responses contain stereotypical content. Incorporating chain-of-thought reasoning prior to generation reduces bias by up to half in some models, though effectiveness varies across architectures. The findings indicate that current quantization strategies fail to adequately preserve bias-mitigation signals, underscoring the necessity of integrating fairness considerations into model compression design.

Abstract

Almost every large language model that reaches a broad audience is quantized: trained in full precision, then compressed for efficiency. This step is assumed harmless and its safety is rarely re-checked. We find its principal side effect is increased bias that standard safety evaluation misses. Holding the model, its training, and the prompts fixed, a quantized model still refuses harmful requests, still avoids over-refusing benign prompts, and still selects the unbiased multiple-choice answer. Yet asked an open-ended question, the same model volunteers stereotypes in all eight languages we probe, in roughly one in four open-ended answers under an independent judge (~24% to ~27% across the compression ladder): it passes every standard check and still reaches users measurably more biased. The selective gap is a robust finding; whether open-ended bias further increases with compression is less certain, sensitive to the judge that scores it. We address both with \textbf{QuantiBias}, a benchmark that pairs a generative, multilingual stereotype probe with the refusal and multiple-choice controls that isolate open-ended generation, contrasts each build with and without reasoning, and rates the content severity of what it generates. Across two backbone models (Qwen and Gemma), a five-family screen, and eight benchmarks, quantizers allocate their extra precision by capability data that carries no bias-prevention signal, and reasoning before answering roughly halves the effect on some families while doing nothing on others. A quantized build must be re-evaluated for open-ended bias, not only on the short-form safeguards it already passes.

Problem

• quantization
• bias
• large language models
• open-ended generation
• safety evaluation

Innovation

• quantization-induced bias
• open-ended generation
• stereotype probing
• multilingual benchmark
• reasoning-augmented evaluation