On the Diverse Dynamical Behaviors Arising in Deep Linear Transformers

2026-07-20

AI Summary

This study investigates the dynamic evolution of deep linear encoder Transformers during inference. By interpreting tokens as particles, the authors model linear self-attention layers as a generalized Kuramoto system in a two-dimensional embedding space through the lens of interacting particle systems. Leveraging Watanabe–Strogatz theory, Ott–Antonsen manifold reduction, and dynamical systems stability analysis, they establish—for the first time—the existence of an intrinsic low-dimensional structure and a hidden Hamiltonian structure underlying these dynamics. The work rigorously proves the presence and structural stability of long-term behaviors such as clustering, oscillations, and bifurcations in the two-dimensional setting. Numerical experiments further demonstrate that these phenomena persist prominently in higher-dimensional Transformer architectures.

Abstract

We study the inference-time behavior of deep linear encoder-only transformers through the lens of interacting particle systems. In this perspective, tokens are modeled as particles that interact dynamically through successive linear self-attention layers. We show that in embedding dimension two, for any key, query, and value matrices, the dynamics can be reformulated as a generalized Kuramoto-type model with pure second-harmonic coupling. This formulation is amenable to Watanabe--Strogatz theory which reveals the dynamics are intrinsically low-dimensional regardless of the parameter matrices. For a class of token initializations associated with the Ott--Antonsen (OA) manifold, we show that the parameter matrices induce a diverse variety of long-time behaviors in linear transformers, including clustering, oscillations, and bifurcations. The oscillations and bifurcations are characterized by uncovering a hidden Hamiltonian structure in the dynamics. By establishing a structural stability result, we further show that dynamics initialized near the OA manifold exhibit the same long-time behavior as those initialized exactly on the manifold. Motivated by our theory in dimension two, we conduct numerical experiments for analogous parameter regimes in higher-dimensional transformers. Our numerical experiments suggest that the long-time behaviors characterized in our theoretical results persist in higher dimensions.

Problem

• linear transformers
• dynamical behaviors
• Kuramoto model
• Ott–Antonsen manifold
• Hamiltonian structure

Innovation

• linear transformers
• interacting particle systems
• Kuramoto model
• Ott–Antonsen manifold
• Hamiltonian dynamics