Temporal Dependencies in In-Context Learning: The Role of Induction Heads

2026-04-01

AI Summary

It remains unclear how large language models process temporal information during in-context learning. Drawing on the free recall paradigm from cognitive science, this work systematically investigates the mechanisms by which these models capture sequential dependencies and establishes, for the first time, a causal link between induction heads and the pervasive +1 lag bias observed in in-context learning. Through attention head ablation, quantification of induction scores, and evaluation on few-shot sequence recall tasks, the study demonstrates that removing attention heads with high induction scores significantly attenuates the +1 lag bias and impairs recall performance, whereas random ablation produces no such effect. These findings confirm the critical role of induction heads in the ordered retrieval of temporally structured information.

Technology Category

Natural Language Processing: (Large) Language ModelsCognitive Modeling & Cognitive Systems: Conceptual Inference and ReasoningMachine Learning: Causal Learning

Application Category

Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for rankingUser Modeling, Personalization and Recommendation: Fairness-aware retrieval and rankingSemantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Large language models (LLMs) exhibit strong in-context learning capabilities, but how they track and retrieve information from context remains underexplored. Drawing on the free recall paradigm in cognitive science (where participants recall list items in any order), we show that several open-source LLMs consistently display a serial-recall-like pattern, assigning peak probability to tokens that immediately follow a repeated token in the input sequence. Through systematic ablation experiments, we show that induction heads, specialized attention heads that attend to the token following a previous occurrence of the current token, play an important role in this phenomenon. Removing heads with a high induction score substantially reduces the +1 lag bias, whereas ablating random heads does not reproduce the same reduction. We also show that removing heads with high induction scores impairs the performance of models prompted to do serial recall using few-shot learning to a larger extent than removing random heads. Our findings highlight a mechanistically specific connection between induction heads and temporal context processing in transformers, suggesting that these heads are especially important for ordered retrieval and serial-recall-like behavior during in-context learning.

Problem

• in-context learning
• temporal dependencies
• induction heads
• serial recall
• attention mechanisms

Innovation

• induction heads
• in-context learning
• temporal dependencies
• serial recall
• attention mechanism