An Algebraic View of the Expressivity of Recurrent Language Models

2026-06-01

AI Summary

This work resolves theoretical discrepancies regarding the capacity of recurrent neural language models to recognize formal languages under different arithmetic models, offering a unified characterization of their expressive power. By constructing a framework grounded in algebraic automata theory, the study reformulates the question of language expressivity as an algebraic problem: whether the syntactic monoid of a language divides a specific wreath product. The analysis reveals that the choice of quantization critically affects computational capability—under floating-point recurrence, diagonal state-space models cannot implement even-modulo counters, whereas with unsigned integer quantization, they can realize all such counters. This result provides a novel algebraic perspective on the theoretical limits of recurrent architectures.

Abstract

What formal languages can a recurrent neural language model recognize? Formal results in the literature conflict: some authors report Turing-completeness, while others show equivalence to regular languages. The reason for this discrepancy is that the underlying arithmetic model differs. The paper develops a unified algebraic account of the expressivity of recurrent neural networks, starting with a formal account of various arithmetic models. This account reduces expressivity to an algebraic question, e.g., whether a network's syntactic monoid divides a certain wreath product. As a case study, the paper revisits diagonal state-space models: the same architecture cannot implement an even-modulus counter once floating-point recurrences are enforced, yet realizes every even-modulus counter under unsigned-integer quantization.

Problem

• recurrent language models
• formal languages
• expressivity
• arithmetic models
• Turing-completeness

Innovation

• algebraic expressivity
• recurrent neural networks
• arithmetic models
• syntactic monoid
• wreath product