Pause Tokens Strictly Increase the Expressivity of Constant-Depth Transformers

2025-05-27

AI Summary

Prior empirical work observed that “pause tokens” (e.g., “...”) improve the performance of shallow Transformers, yet their theoretical role remains unexplained. Method: We formalize this phenomenon using circuit complexity theory, modeling bounded- or logarithmic-precision activations, causal masking, and polynomially many pause-token insertions. Contribution/Results: We prove that constant-depth, logarithmic-width Transformers with pause tokens gain strictly greater computational power: they upgrade from computing only a proper subset of $mathsf{AC}^0$ to computing all of $mathsf{AC}^0$ (under bounded precision) or $mathsf{TC}^0$ (under logarithmic precision)—the first rigorous complexity-theoretic separation for Transformers with pause tokens. Empirical validation via parity learning confirms this: a two-layer Transformer succeeds on the parity function only when pause tokens are present; it fails completely without them. Our analysis reveals how pause tokens alleviate expressivity bottlenecks under depth constraints and elucidates their synergistic interplay with width and numerical precision.

Technology Category

• Machine Learning: Deep Neural Architectures and Foundation Models
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning
• Search and Optimization: Learning to Search

Application Category

• Web Mining and Content Analysis: Large pretrained models with web data
• Security and Privacy: Privacy-enhancing technologies
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Pause tokens, simple filler symbols such as"...", consistently improve Transformer performance on both language and mathematical tasks, yet their theoretical effect remains unexplained. We provide the first formal separation result, proving that adding pause tokens to constant-depth, logarithmic-width Transformers strictly increases their computational expressivity. With bounded-precision activations, Transformers without pause tokens compute only a strict subset of $mathsf{AC}^0$ functions, while adding a polynomial number of pause tokens allows them to express the entire class. For logarithmic-precision Transformers, we show that adding pause tokens achieves expressivity equivalent to $mathsf{TC}^0$, matching known upper bounds. Empirically, we demonstrate that two-layer causally masked Transformers can learn parity when supplied with pause tokens, a function that they appear unable to learn without them. Our results provide a rigorous theoretical explanation for prior empirical findings, clarify how pause tokens interact with width, depth, and numeric precision, and position them as a distinct mechanism, complementary to chain-of-thought prompting, for enhancing Transformer reasoning.

Problem

• Pause tokens enhance Transformer expressivity theoretically unexplained
• Formally prove pause tokens increase computational expressivity in Transformers
• Pause tokens enable Transformers to learn previously unattainable functions

Innovation

• Pause tokens enhance Transformer expressivity theoretically
• Pause tokens enable Transformers to compute broader function classes
• Pause tokens improve learning parity in shallow Transformers