Mitigating Hallucinations in Large Language Models Via Decoder Layer Skipping

2026-05-30

AI Summary

Large language models often generate factually hallucinated content due to deep decoder architectures. This work proposes DeLask, a novel decoding framework that uniquely integrates dynamic layer-skipping with gradient descent direction consistency. By computing the cosine similarity—termed “driftance”—between each decoder layer’s hidden state and the ideal update direction, DeLask dynamically skips layers prone to inducing hallucinations while partially fusing their hidden states to preserve semantic coherence. The approach is lightweight and model-agnostic, achieving significant reductions in hallucination rates across multiple mainstream large language models and standard benchmarks, all while maintaining strong expressiveness and high-quality text generation.

Abstract

Large Language Models (LLMs) have achieved strong performance across diverse natural language tasks, yet their outputs often suffer from hallucinations -- content that is misaligned with factual information. In this work, we conduct a comprehensive layer-wise analysis of the decoding process and reveal that hallucinations tend to originate from deeper decoder layers. To address this issue, we introduce \textbf{DeLask} (\textbf{De}coder \textbf{La}yer \textbf{Sk}ipping), a novel decoding framework that dynamically skips layers prone to producing hallucinations. DeLask leverages the theoretical insight that the forward computation of an $L$-layer Transformer is conditionally equivalent to $L$ steps of gradient descent. We define a \emph{driftance value} by computing the cosine similarity between gradients derived from consecutive decoder steps, identifying problematic layers when the descent direction reverses. Rather than discarding such layers entirely, DeLask partially aggregates their hidden states with preceding layers, thereby preserving consistency while suppressing erroneous signals. Extensive experiments across diverse LLMs and benchmarks demonstrate that DeLask consistently mitigates hallucinations and enhances overall reliability, providing a lightweight and generalizable decoding framework for improving the robustness of large-scale language models.

Problem

• Hallucinations
• Large Language Models
• Decoder Layers
• Factual Consistency
• Reliability

Innovation

• hallucination mitigation
• decoder layer skipping
• gradient descent analogy
• driftance value
• consistent decoding