MemoReason: Evaluating the Effect of Parametric Memory on Contextual Reasoning in LLMs

2026-09-28

AI Summary

This study investigates whether the reasoning performance of large language models stems from genuine in-context reasoning or reliance on parametric memory. To address this, we propose a paired evaluation framework that employs structured text substitution to replace benchmark entities with fictitious items, thereby precisely disentangling and quantifying memory and reasoning effects while preserving task structure. Experimental results demonstrate that model performance degrades by up to 15.7% under fictitious scenarios, confirming the presence of parametric memory bias while ruling out direct parameter shortcuts as the primary failure mode. This work establishes a novel paradigm for the objective assessment of the authentic reasoning capabilities of large language models.

Abstract

Large Language Models (LLMs) perform well on reasoning benchmarks, but it remains unclear whether this reflects genuine contextual reasoning or reliance on facts memorized in their parameters. We investigate this by distinguishing two possibilities: a broad \textit{memorization bias}, where familiar content improves reasoning performance, and the \textit{Strong Parametric Shortcut Hypothesis}, where models skip reasoning entirely and recall stored answers. To test these effects, we introduce \textbf{MemoReason}, a human-curated benchmark that pairs factual reasoning tasks with structurally identical \fictitiousterm{} versions where real entities like people, companies, or dates are systematically replaced by \fictitiousterm{} ones of the same type. This \scorerevision{preserves task structure and specified reasoning operations} while varying the familiarity of the context, allowing controlled measurement of how the parametric memory affects reasoning. \revision{Our evaluation of recent LLMs reveals consistent and statistically significant performance drops of up to 15.7\% in the fictitious setting, demonstrating a clear memorization bias.} However, a targeted analysis of \revision{questions failed in the fictitious setting} shows that models rarely respond with the corresponding factual answer, indicating that direct parametric shortcuts are not the dominant failure mode. These findings suggest that parametric memory influences reasoning through mechanisms more complex than simple factual recall. \textbf{MemoReason} provides a controlled framework for studying these mechanisms and for extending paired factual-fictitious{} evaluation to broader reasoning settings.

Problem

• Large Language Models
• Contextual Reasoning
• Parametric Memory
• Memorization Bias
• Parametric Shortcut

Innovation

• Parametric Memory
• Contextual Reasoning
• MemoReason Benchmark
• Fictitious Entities
• Memorization Bias