Inside the LLM Word Factory

2026-06-07

AI Summary

This study investigates how large language models (LLMs), which operate on subword units, internally aggregate these into word-level representations—a process critical for semantic understanding yet poorly understood. Through activation patching, controlled paired experiments, and probing analyses across twelve prominent models including Llama2-7B, the work identifies a universal two-stage detokenization mechanism: early attention layers propagate subword signals, followed by multilayer perceptrons (MLPs) that fuse local embeddings into coherent word representations. This mechanism is consistently observed across eight model families and is modulated by positional encoding schemes—emerging in layers 1–5 in models using RoPE, but delayed to layers 5–10 in those with absolute positional encodings. Leveraging these insights, an early-layer activation probe achieves AUROC scores of 0.94–0.97 on English linguistic tasks.

Abstract

Transformer language models process input provided as subword fragments, but natural language semantics usually rely on word-level concepts. Detokenization is the process where models reconcile these two facts, aggregating subwords into word-level representations through their computation. Prior work has found that this takes place mostly in early-to-middle layers, but so far the exact mechanics of the process have not been pinned down. We venture deep into detokenization using activation patching in controlled paired experiments that isolate the contribution of different model components, localizing English detokenization in Llama2-7B to a two-stage process at Layer 1. Attention transmits a token-specific signal from nonfinal subwords, using sequential relays if necessary, while the MLP composes it with the local embedding. This two-stage structure generalizes to twelve models from eight families, but the depth over which it takes place depends on the flavor of positional encoding: RoPE-based models detokenize over 1 to 5 layers, while learned-absolute models take 5 to 10. Finally, we provide a probe for determining the success of the detokenization process based on early-layer activations alone, performing at 0.94-0.97 AUROC depending on the amount of context.

Problem

• detokenization
• subword
• word-level semantics
• language models
• positional encoding

Innovation

• detokenization
• activation patching
• subword composition
• positional encoding
• MLP-attention interaction