Rethinking the Role of Positional Encoding: Sliding-Window Transformers without PE Remain Turing Complete

2026-05-31

AI Summary

This work investigates whether positional encoding is essential for Transformers to achieve universal computational capability, with a focus on sliding-window inference scenarios. By constructing a positional-encoding-free sliding-window Transformer, the authors introduce the HIST abstract autoregressive framework and rigorously demonstrate—through constant-size internal states, token-count histogram analysis, and Post machine simulation—that the sliding-window mechanism alone suffices to break input permutation symmetry and endow the model with Turing completeness. This finding challenges the prevailing assumption that explicit positional encodings are indispensable, revealing instead that the sliding window itself inherently provides sufficient positional information to support universal computation.

Abstract

Positional encoding (PE) is widely viewed as necessary for transformers to process ordered sequences: without them, the next-token map appears permutation-invariant in its context tokens. This intuition underlies all prior universality results, which rely on positional information to prove that transformers with chain-of-thought can perform arbitrary computation, i.e., they are Turing complete. We revisit this belief in the regime most relevant to long-form reasoning, where generation proceeds through a finite sliding context window. Our opening perception is that the window mechanism itself (mildly) breaks the permutation symmetry. To distill and precisely capture the degree of this added expressiveness, we introduce an abstract autoregressive model, the HIST model, in which each update depends only on constant-size internal state and the token-count histogram within the current window. We prove that this HIST model is Turing complete by showing that the evolution of the window can reveal the token that has just left the window, which suffices to simulate Turing-complete Post machines. We then construct a sliding-window transformer over a constant-size token alphabet, without PE, and show that it can simulate the HIST model. Our result demonstrates that positional encodings are not indispensable for transformers to perform universal computation: The window sliding itself already breaks permutation symmetry and captures sufficient positional information.

Problem

• Positional Encoding
• Transformer
• Turing Completeness
• Sliding Window
• Permutation Invariance

Innovation

• positional encoding
• sliding-window transformer
• Turing completeness
• permutation symmetry
• HIST model