Resonant Context Anchoring: Decoupling Attention Routing and Signal Gain at Inference Time

2026-06-01

AI Summary

This work addresses the tendency of large language models to generate factually hallucinated outputs when input evidence conflicts with internal parametric knowledge, often leading to disregard of contextual information. The authors propose a lightweight, training-free intervention applied at inference time that, without altering attention distributions, orthogonally decouples the routing logic from the information magnitude in self-attention. By leveraging pre-softmax attention scores to construct a dynamic gain field, the method selectively amplifies the norm of value vectors corresponding to contextually relevant tokens, thereby enhancing the signal-to-noise ratio of external evidence in the residual stream. Evaluated on Llama-3 family models, this approach incurs minimal computational overhead and significantly improves performance across multiple factual consistency benchmarks, effectively suppressing parametric hallucinations while achieving a Pareto improvement in both contextual faithfulness and generation fluency.

Abstract

Large Language Models (LLMs) frequently exhibit "contextual disregard" when faced with input evidence that conflicts with their internal parametric memory, leading to persistent factual hallucinations. Existing mitigation strategies primarily rely on suppressing specific neuron activations or employing computationally expensive contrastive decoding mechanisms, which often result in increased perplexity or significantly elevated inference latency. To address these limitations, we propose Resonant Context Anchoring (RCA), a lightweight inference-time intervention method grounded in the perspective of residual stream signal dynamics. RCA aims to resolve the signal attenuation of external evidence during its propagation through deep networks. The core mechanism involves the orthogonal decoupling of routing logic and information magnitude within the self-attention module. By utilizing raw pre-softmax attention scores as an instantaneous metric of semantic alignment, we construct a dynamic gain field via non-linear rectification to selectively amplify the norms of value vectors corresponding to context tokens, without altering the attention probability distribution. This mechanism effectively elevates the signal-to-noise ratio (SNR) of input evidence within the residual stream mixture, thereby robustly anchoring the generation trajectory to the truthful context during inference. Extensive experiments on the Llama-3 model series demonstrate that RCA significantly improves contextual faithfulness across multiple factual consistency and strong knowledge-conflict tasks, effectively suppressing parametric hallucinations. Furthermore, results confirm that as a training-free and computationally negligible plug-and-play module, RCA achieves a Pareto improvement in faithfulness and fluency while maintaining the model's general language understanding capabilities.

Problem

• contextual disregard
• factual hallucinations
• signal attenuation
• residual stream
• self-attention

Innovation

• Resonant Context Anchoring
• attention decoupling
• signal gain
• inference-time intervention
• hallucination suppression