TextNCA: Neural Cellular Automata for Language Modeling via Hierarchical Local Attention

2026-08-03

AI Summary

This work investigates whether strictly local, iterative, and weight-shared computational primitives can effectively support language modeling, and disentangles the individual contributions of locality, iteration, and weight sharing. To this end, the authors propose TextNCA, the first architecture applying neural cellular automata to language modeling, which employs a hierarchical causal sliding window (8/32/128), multi-step iterative updates with shared weights, GRU gating, and learnable step-wise embeddings. Experiments reveal that hierarchical window scheduling is critical for performance, while iteration provides only marginal gains: on WikiText-103, TextNCA achieves 60.3 perplexity with approximately 30M parameters in 60k training steps, whereas a sliding-window Transformer without iteration but preserving the scheduling incurs only a +4.1 perplexity penalty, and disrupting the scheduling leads to substantial degradation.

Abstract

Can a strictly local, iterated, weight-shared computation primitive support language modelling, and which of those three properties actually drives the model's behaviour? We define \textsc{TextNCA}, a 1D causal windowed-attention realisation of the Neural Cellular Automaton primitive, and study a hierarchical variant that cascades three stages with windows $w \in \{8, 32, 128\}$ and $T_s$ shared-weight iterations per stage, all on WikiText-103 at roughly 30M parameters and 60k training steps. The model does not match a parameter-matched Transformer at this scale (Hier-TextNCA $60.3$ vs.\ Transformer-6L $52.8$ and Transformer-12L $44.7$ PPL), so we treat it as an analytical probe rather than a proposed alternative. The behaviour we observe is largely explained by the staged narrow-to-wide schedule: a non-iterating sliding-window Transformer that reuses the same schedule comes within $+4.1$ PPL of the iterated model, while reversing, flattening, or breaking the monotonic ordering of the schedule costs between $+16.7$ and $+70.8$ PPL. Iteration adds a smaller bounded benefit on top of the schedule, with a clear optimum at $T_s{=}4$ and a U-shaped degradation beyond it. The GRU gate and learned per-step embeddings are required for that benefit to appear, and training with random $T_s$ yields an inference-time iteration-count knob at the cost of substantially higher absolute PPL. We position the work as a controlled reading of which parts of NCA-style computation carry the weight in language modelling.

Problem

• language modeling
• neural cellular automata
• local attention
• weight sharing
• iterated computation

Innovation

• Neural Cellular Automata
• Hierarchical Local Attention
• Weight-Shared Iteration
• Causal Windowed Attention
• Language Modeling