Length Generalization Bounds for Transformers

2026-02-13

AI Summary

This study investigates the length generalization capability of Transformers—specifically, their ability to generalize to arbitrarily long inputs when trained on sequences of bounded length—and the computability of associated generalization bounds. Leveraging formal language theory and computational complexity analysis, and employing C-RASP, a mathematically rigorous abstraction of the Transformer architecture, the work establishes for the first time that C-RASP models with two or more layers admit no computable length generalization bound. In contrast, both the positive fragment of C-RASP and fixed-precision Transformers possess tight, optimal, and computable exponential generalization bounds. These results uncover fundamental limitations inherent in deep Transformer models regarding length generalization while providing an exact characterization of the conditions under which such bounds remain computable.

Abstract

Length generalization is a key property of a learning algorithm that enables it to make correct predictions on inputs of any length, given finite training data. To provide such a guarantee, one needs to be able to compute a length generalization bound, beyond which the model is guaranteed to generalize. This paper concerns the open problem of the computability of such generalization bounds for CRASP, a class of languages which is closely linked to transformers. A positive partial result was recently shown by Chen et al. for CRASP with only one layer and, under some restrictions, also with two layers. We provide complete answers to the above open problem. Our main result is the non-existence of computable length generalization bounds for CRASP (already with two layers) and hence for transformers. To complement this, we provide a computable bound for the positive fragment of CRASP, which we show equivalent to fixed-precision transformers. For both positive CRASP and fixed-precision transformers, we show that the length complexity is exponential, and prove optimality of the bounds.

Problem

• length generalization
• transformers
• C-RASP
• generalization bounds
• computability

Innovation

• length generalization
• transformers
• C-RASP
• computability
• generalization bounds