🤖 AI Summary
This study investigates whether language models can effectively communicate intermediate reasoning during inference by transmitting hidden activations rather than textual tokens. Focusing on multi-hop reasoning tasks with Pythia models ranging from 160M to 410M parameters, the authors train linear mappings to align normalized hidden states between sender and receiver models and explore injecting the translated activations into the receiver either additively or via replacement. Despite achieving cosine similarities as high as 0.97—indicating strong representational alignment—neither injection strategy improves downstream performance: additive injection yields no significant gains, while replacement consistently degrades it. This work presents the first controlled demonstration of direct cross-model activation transfer and reveals the counterintuitive finding that representational alignment alone is insufficient to enable effective causal communication.
📝 Abstract
Recent work shows that language models can transmit behavioural traits through hidden signals in generated data during training. We ask whether a more direct and stricter channel is also viable: can one language model communicate useful intermediate reasoning state to another at inference time by translating and injecting hidden activations, rather than by passing natural-language text? We test this question in a controlled Pythia-160M to Pythia-410M multi-hop reasoning setting. A linear translation layer learns a strong normalized-space map between sender and receiver hidden states, with normalized cosine similarity near 0.97 across seeds. However, when the translated activations are injected into the receiver at inference time, they do not improve downstream answering. Low-strength additive injection remains near the no-injection baseline, with confidence intervals that cross zero. Replacement-style injection is consistently destructive, and rescaling translated vectors to the receiver hidden-state norm does not rescue performance. The result is therefore a scoped negative result: in this setting, offline representational alignment is not sufficient for useful causal communication inside the receiver.