The Latin Substrate: How Language Models Represent and Mediate Script Choice

2026-05-29

AI Summary

How large language models regulate script selection in multilingual settings remains unclear. This work investigates the underlying mechanisms through layer-wise output distribution analysis, representational geometry, and causal interventions, revealing a systematic representational bias toward Latin scripts. We identify generalizable linear steering directions and cross-lingually shared attention heads that elucidate the asymmetric nature of script generation. Leveraging logit lens analysis, representational similarity metrics, and linear steering techniques, we pinpoint critical network components enabling semantically consistent cross-script transformation: non-Latin scripts can be efficiently converted into Latin scripts via compact gating mechanisms, whereas the reverse transformation relies on distributed representations.

Abstract

Many languages are written in multiple scripts, requiring large language models (LLMs) to generate equivalent linguistic content in distinct orthographic forms. While prior work suggests that LLMs route information through shared latent representations, how they internally mediate script variation remains poorly understood. We study this question by first examining per-layer output distributions with the logit lens, which reveals consistent latent romanization during transliteration, and then through representational and mechanistic analyses of script generation. At the representational level, we show that scripts of the same language become increasingly separable across layers and that a simple linear steering direction can flip a model's output script while largely maintaining semantic content. The vector generalizes asymmetrically to writing systems unseen during construction, flipping non-Latin output to Latin reliably, but mapping Latin output into varied non-Latin scripts. At the mechanistic level, we localize a small set of late-layer attention heads that causally mediate script choice. These heads transfer across unrelated languages and writing systems, suggesting that script routing is implemented by language-agnostic components. Across both analyses, we observe a consistent directional asymmetry: non-Latin output is produced by a compact, identifiable gate, while Latin-script output emerges from diffuse contributions across the network. Collectively, our findings hint that LLMs organize script variation around shared latent representations while exhibiting a privileged substrate toward Latin script.

Problem

• script choice
• language models
• latent representations
• transliteration
• Latin script

Innovation

• latent romanization
• linear steering
• attention heads
• script mediation
• privileged substrate