Technology Category
Cognitive Modeling & Cognitive Systems: AnalogyNatural Language Processing: (Large) Language ModelsMachine Learning: Large Multimodal Models (LMMs)
Emergence of Episodic Memory in Transformers: Characterizing Changes in Temporal Structure of Attention Scores During Training
🤖 AI Summary
This study investigates the temporal evolution of attention scores during Transformer training and its impact on transient information organization in in-context learning, specifically examining parallels with and distinctions from human episodic memory phenomena—including temporal proximity effects, primacy/recency effects, and serial recall biases. Using multi-scale GPT-2 models, we integrate cognitive psychology experimental paradigms, attention-head interpretability analysis, targeted ablation, and sequence-level behavioral modeling. We provide the first systematic causal validation that “induction heads” constitute the core mechanism driving episodic-memory-like temporal biases: their ablation fully eliminates proximity effects; all model scales consistently exhibit human-like temporal memory patterns, with serial recall accuracy significantly exceeding random baselines. These findings establish a rigorous interdisciplinary bridge between cognitive science and large language model representation analysis, advancing mechanistic understanding of how neural sequence models emulate structured temporal cognition.
Technology Category
Application Category
📝 Abstract
We investigate in-context temporal biases in attention heads and transformer outputs. Using cognitive science methodologies, we analyze attention scores and outputs of the GPT-2 models of varying sizes. Across attention heads, we observe effects characteristic of human episodic memory, including temporal contiguity, primacy and recency. Transformer outputs demonstrate a tendency toward in-context serial recall. Importantly, this effect is eliminated after the ablation of the induction heads, which are the driving force behind the contiguity effect. Our findings offer insights into how transformers organize information temporally during in-context learning, shedding light on their similarities and differences with human memory and learning.
Problem
Research questions and friction points this paper is trying to address.
Transformers' episodic memory emergence
Temporal structure in attention scores
Comparison with human memory processes
Innovation
Methods, ideas, or system contributions that make the work stand out.
Analyzes GPT-2 attention scores
Links transformers to episodic memory
Identifies induction heads' temporal role
🔎 Similar Papers
No similar papers found.
💼 Related Jobs
Indiana University Bloomington
