FIDES: Faithful Inference via Deep Evidence Signals for Retrieval-Memory Conflict in RAG

2026-06-03

AI Summary

This work addresses the issue in retrieval-augmented generation (RAG) where language models often disregard retrieved context due to conflicts between their parametric memory and external evidence, leading to factually distorted outputs. The authors propose a training-free dynamic contrastive decoding method that identifies— for the first time—that such conflicts predominantly occur at a small number of critical decoding steps. Leveraging this insight, they design a conflict-aware mechanism that integrates output logits, hidden representations, and prediction trajectories to enable on-demand, token-level intervention. Evaluated across three benchmarks and six backbone models ranging from 7B to 70B parameters (18 settings in total), the approach consistently improves contextual faithfulness, achieving 92–94% faithfulness and 62–63% F1 scores on the 70B-scale models.

Abstract

When retrieved evidence contradicts parametric memory, language models frequently ignore context and default to memorized priors -- a failure that undermines the core purpose of retrieval augmentation. Contrastive decoding amplifies the context-conditioned output to suppress parametric bias, but existing methods rest on an implicit assumption that this bias is uniform across tokens. A single global contrastive weight over-penalizes safe tokens while leaving genuinely conflicted ones insufficiently corrected. We identify token-level conflict concentration: retrieval-memory tension is sharply heterogeneous, concentrated on a small fraction of answer-critical decoding steps. This reframes contrastive decoding from how much contrast to apply to where to apply it. We propose FIDES (Faithful Inference via Deep Evidence Signals), a training-free decoder that reads three internal signals probing retrieval-memory conflict at complementary depths -- output surface, hidden representations, and prediction trajectory -- and fuses them to govern intervention strength at each decoding step. Across three benchmarks and six backbones -- four primary 7B/8B models and two scaling backbones up to 70B -- FIDES achieves the best context fidelity in all 18 settings, outperforming the strongest training-free baseline by +3 to +13 points. On the 70B scale, fidelity reaches 92-94% while F1 surges to 62-63%, demonstrating that token-level selectivity unlocks generation capability that coarse contrastive rules suppress.

Problem

• retrieval-memory conflict
• contrastive decoding
• context fidelity
• parametric bias
• token-level heterogeneity

Innovation

• contrastive decoding
• retrieval-augmented generation
• token-level conflict
• training-free inference
• evidence fusion