Memory vs. Context? Influential Factors of Factual Recall in Language Models

2026-09-21

AI Summary

研究通过在31个模型上复现和压力测试，探讨语言模型如何在记忆知识和上下文陈述间抉择，揭示了模型大小、实体频率等因素的影响。

Abstract

We reproduce and stress-test the work of Yu et al. (2023), who characterize how language models (LMs) arbitrate between memorized knowledge and contradictory in-context statements. We replicate their world-capitals experiments on 31 models spanning Pythia, GPT-2, Qwen3, and Ministral families, including base and post-trained variants, and extend evaluations to five additional knowledge relation types from the ParaConflict dataset. We empirically confirm most of their original findings: larger models and higher-frequency entities tend to favor memorized answers, with substantial family-level variance. However, several conclusions do not generalize cleanly: entity-frequency effects disappear on Qwen3-14B and 32B; post-training shifts the memory-context trade-off inconsistently across families; question phrasing alone can change a model's reliance on memorized knowledge by up to 80 percentage points; and semantically unrelated prose can mimic coherent supporting context. Our results clarify where Yu et al.'s claims hold and to what extent they generalize to other prompts.

Problem

• Language Models
• Factual Recall
• Memorized Knowledge
• In-Context Statements

Innovation

• language models
• knowledge arbitration
• memory vs. context
• post-training effects
• question phrasing