A Compositional Theory of Causally Masked Transformers

2026-07-29

AI Summary

This study investigates the expressive power of causal-masked Transformers with finite-precision arithmetic in solving decision problems over arbitrarily long input sequences. By integrating algebraic formalism with finite-precision numerical semantics and semigroup theory, the authors develop a compositional framework centered on memory states to analyze representational capacity. They establish, for the first time, a precise correspondence between four classes of attention mechanisms and specific semigroup varieties—namely, aperiodic, R-trivial, locally R-trivial, and aperiodic semigroups—and prove that, under the free wiring assumption, the expressiveness boundaries of these classes are all tight. This work offers a novel algebraic perspective for understanding the theoretical capabilities of Transformers under realistic computational constraints.

Abstract

What types of decision problems can a causally masked, finite-precision transformer solve for inputs of arbitrary length? Existing answers often rely on idealized arithmetic, but under finite precision, rounding and evaluation order can change what information attention retains and therefore what the model can compute. We develop an algebraic formalization that derives expressivity directly from the model's implemented dynamics. Its central object is its memory; the finite internal state computed by attention that summarizes the information from the prefix available to all future queries. Each attention head updates its own state independently within a layer, while layers compose hierarchically, providing a uniform route from model assumptions to expressivity bounds. Applying this method to transformers without positional embeddings, we obtain an expressivity hierarchy governed by the attention type under specific numerical semantics. Width-one sliding-window attention supports bounded-suffix memory, while a modified form of soft attention supports irreversible, checklist-like state, and combining the two mechanisms provides an interplay of both. Ordinary left-to-right floating-point soft attention can realize more expressive memory operations than any of the above. Algebraically, the four cases correspond to definite, R-trivial, locally R-trivial, and aperiodic semigroups. Under an explicit free-wiring assumption, all four bounds are tight.

Problem

• causally masked transformers
• finite precision
• expressivity
• attention mechanisms
• decision problems

Innovation

• compositional theory
• finite-precision transformers
• causal masking
• memory expressivity
• algebraic formalization