Hallucinations as Orthogonal Noise: Inference-Time Manifold Alignment via Dynamic Contextual Orthogonalization

2026-06-01

AI Summary

This work addresses the reliability challenges posed by hallucinations in large language models, which often generate content inconsistent with contextual facts or logic. It introduces a novel geometric perspective, interpreting hallucinations as orthogonal noise on the semantic manifold of the residual stream, and proposes Dynamic Context Orthogonalization (DCO)—a training-free, inference-time intervention. DCO treats the input residual stream as a dynamic anchor, performs orthogonal decomposition of attention head outputs, and suppresses anomalous orthogonal components via a layer-wise Z-score mechanism to align representations with the semantic manifold. Evaluated on Llama-3-8B and Llama-3-70B, DCO significantly improves contextual faithfulness on benchmarks such as XSum, NQ-Swap, and IFEval, while maintaining strong performance on knowledge-intensive tasks like TriviaQA and TruthfulQA, thereby overcoming the typical trade-off between hallucination suppression and knowledge retention.

Abstract

Hallucination in Large Language Models (LLMs), characterized by the generation of content inconsistent with contextual facts or logical constraints -- remains a persistent challenge for reliable deployment. In this work, we address this issue through a geometric framework rooted in the linear representation hypothesis. We propose that hallucinations manifest as orthogonal noise relative to the semantic manifold of the residual stream. Specifically, we hypothesize that while attention heads ideally propagate information congruent with the context subspace, hallucinations arise when specific heads introduce components orthogonal to this subspace, disrupting the coherence of the latent representation. Based on this formulation, we introduce Dynamic Contextual Orthogonalization (DCO), an inference-time intervention method. DCO utilizes the input residual stream as a dynamic context anchor to perform orthogonal decomposition on attention head outputs. To distinguish between context-aligned semantic updates and divergent noise, DCO employs a layer-wise Z-score suppression mechanism that selectively attenuates outlier orthogonal components based on statistical distributions. Evaluations on Llama-3-8B and 70B across benchmarks such as XSum, NQ-Swap, and IFEval demonstrate that DCO achieves superior contextual faithfulness compared to state-of-the-art intervention baselines. Furthermore, DCO maintains high performance on knowledge-intensive tasks like TriviaQA and TruthfulQA, effectively mitigating the trade-off between hallucination suppression and parametric knowledge retention often observed in existing methods. Our findings validate the geometric interpretation of hallucinations and establish DCO as a computationally efficient approach for enforcing manifold alignment.Our code is available at https://github.com/Harry-Miral/DCO

Problem

• Hallucination
• Large Language Models
• Contextual Faithfulness
• Semantic Manifold
• Inference-Time Intervention

Innovation

• hallucination
• manifold alignment
• orthogonal noise
• inference-time intervention
• dynamic contextual orthogonalization