The Alignment Paradox: How Post-Training Amplifies Confident Hallucinations in Language Models

2026-09-26

AI Summary

This study addresses the paradox whereby post-training alignment of large language models induces a surge in high-confidence hallucinations. Through layer-wise probing with Logit Lens, we reveal that the alignment process amplifies model confidence in erroneous outputs. To mitigate this issue, we propose an entropy-dependent Direct Preference Optimization (DPO) margin constraint mechanism that restricts the excessive expansion of decision boundaries in the model's later layers, thereby fundamentally suppressing the generation of confident hallucinations. Experiments on Mistral-7B demonstrate that our approach significantly reduces high-confidence errors by 35.3% while effectively preserving general reasoning capabilities. This work establishes a novel paradigm for alleviating alignment-induced hallucinations without compromising overall model utility.

Abstract

Large language models (LLMs) can produce factually incorrect answers with high confidence, undermining their reliability and limiting the effectiveness of uncertainty-based error detection. While prior research attributes confident hallucinations to factors such as missing knowledge in training data, reasoning errors, or stochastic decoding, we uncover that post-training alignment itself is a primary driver of these errors, a phenomenon we call the \textbf{Alignment Paradox}. Across five model families evaluated on factual benchmarks, unaligned base models produce few high-confidence errors on long-tail factual queries, whereas instruction-tuned models multiply high-confidence errors ($p \ge 0.95$) by more than an order of magnitude (10$\times$ to 35$\times$). Layer-wise probing with the Logit Lens reveals that this overconfidence emerges in late layers, where wrong-answer margins expand past 4.0 points after remaining near zero across early and intermediate layers. These findings motivate limiting margin growth during post-training. We implement this principle through an entropy-dependent margin bound in direct preference optimization (DPO). In multi-epoch experiments with Mistral-7B, the bounded objective reduces high-confidence errors by up to 35.3\% relative to standard DPO while maintaining performance on evaluated general reasoning benchmarks. These results show that bounded margins mitigate confident hallucinations during post-training.

Problem

• confident hallucinations
• alignment paradox
• large language models
• post-training alignment
• overconfidence

Innovation

• Alignment Paradox
• Confident Hallucinations
• Direct Preference Optimization
• Entropy-Dependent Margin Bound
• Logit Lens