VC Dimension and Expressivity of Real-Valued Transformers

2026-09-25

AI Summary

This study investigates the theoretical expressivity and generalization bounds of assumption-free, real-valued Transformers. Overcoming the limitations of prior simplified models, it leverages real algebraic geometry to conduct VC dimension analysis and constructive model design, deriving tight upper and lower complexity bounds. The research demonstrates that the VC dimension of real-valued Transformers is O(n⁴), revealing their expressive limits regarding symmetric functions as well as the constraints imposed by finite numerical precision. This work establishes the first rigorous generalization theoretical framework for general real-valued Transformers, providing a precise quantitative characterization of their expressive capacity.

Abstract

Whereas previous results on abilities and limitations of transformers have restricted the definition of transformers in various ways, here we study softmax-attention, multi-layer transformers operating on real values, with very few additional assumptions. Applying results from real geometry, we obtain upper bounds on the VC dimension and split VC dimension of such transformers ($O(n^4)$ and $O(n^6)$, respectively, where $n$ is the input length). Conversely, we also construct specific transformers witnessing lower bounds on these quantities ($\Omega(n)$ in each case). These results have some notable consequences. For example, within the class of symmetric (permutation-invariant) functions, we show that transformers can uniformly express all functions over an alphabet of one symbol and non-uniformly express all functions over an alphabet of two symbols, but cannot (even non-uniformly) express some functions over an alphabet of six symbols. We also prove limitations on how many bits of a real number a transformer can access.

Problem

• VC dimension
• expressivity
• real-valued transformers
• softmax attention
• symmetric functions

Innovation

• VC Dimension
• Expressivity
• Real-Valued Transformers
• Softmax Attention
• Real Geometry