A retrieval conditioned rebinding circuit for dynamic entity tracking in large language models

2026-06-07

AI Summary

Large language models struggle to correctly bind entities to their attributes and update these bindings as dynamic states evolve. This work addresses this limitation by revealing, for the first time, a retrieval-condition-triggered “rebinding circuit” that restores and swaps binding information during the readout phase. Through causal intervention, attention head analysis, and vector space dissection, the study demonstrates that this mechanism is prevalent across both Gemma and Llama model families, yet they differ markedly in how binding representations are structurally implemented: Gemma relies on explicit signals within specific query/key subspaces, whereas Llama employs a more distributed encoding strategy. These findings provide an interpretable mechanistic foundation for understanding how large language models track context-dependent state changes.

Abstract

To interpret context correctly and retrieve relevant information, large language models must bind entities to their attributes and update these bindings as state changes. We analyze how LLMs implement this binding process in a dynamic state tracking. Using causal interventions, we identify a retrieval conditioned rebinding mechanism, a compact attention head circuit that encodes swap relevant binding information and reinstates it at readout. Across Gemma and Llama models, this circuit supports rebinding behavior, but the representational signature of the mechanism differs across model families. In Gemma models, the binding signature is clearly expressed in the query/key subspaces of the relevant attention heads, whereas in Llama models, the binding information is carried primarily in key vectors. Overall, our results reveal an interpretable mechanism for context dependent state tracking in LLMs.

Problem

• entity binding
• dynamic state tracking
• large language models
• context interpretation
• attribute updating

Innovation

• retrieval conditioned rebinding
• dynamic entity tracking
• attention head circuit
• binding mechanism
• large language models