Innovation
Methods, ideas, or system contributions that make the work stand out.
Hybrid Language Models
Two-hop Retrieval
Hidden State Intervention
Recurrent Memory
Key Handoff
The Key Handoff: Retrieval in Hybrid Language Models
🤖 AI Summary
This study investigates how entity retrieval and attention mechanisms cooperate within hybrid language models. By conducting hidden state intervention experiments across twelve dense and hybrid architectures, we perform cross-story hidden state transfer to reveal a "critical handoff" mechanism underlying two-hop reasoning. Our findings demonstrate that attention layers are responsible for transforming key information, while recurrent layers maintain and store queryable memory, together forming an efficient division of labor. Notably, modifying the subsequent recurrent memory increases the probability of the target answer by 1.3 to 2.7 times. This work introduces the concept of the "critical handoff" for the first time, explicitly characterizing the hierarchical specialization of retrieval mechanisms in hybrid architectures and offering a novel perspective on their reasoning capabilities.
📝 Abstract
A two-hop question makes a language model retrieve twice: once to produce a bridge entity, and once to retrieve with it. Transformers resolve that entity in their early layers. Hybrid models replace most of the attention with a recurrent state, so where the key becomes usable, and where it is spent on an answer, is not known. Answering"Where is the ball?"from"the ball belongs to Alice"and"Alice is in the garden"turns on Alice, a name the question does not mention. A model could reach garden through Alice, the key it computed, or through where the fact sits in the prompt. Across twelve models, dense and hybrid, we move a hidden state from one story into another where the two routes lead to different places, and read off which place the model gives. An attention layer converts the key in every model we tested, and crossing that layer removes its usable effect, all of it where no attention follows. In sequential hybrids this makes the answer a handoff: recurrent layers carry the key forward, and attention spends it. Retrieval does not always stop there. Writing a different fact into the memory of the recurrent layers that follow the last attention layer can move the answer toward that fact, multiplying its odds by 1.3 to 2.7, and which hybrids do this is not settled by their architecture or training. That read is addressed by the key, not by position: recurrent state in a hybrid is not only a carrier, but a memory that later layers can query.
🔎 Similar Papers
No similar papers found.
