Do Transformers Have the Ability for Periodicity Generalization?

2026-01-30

AI Summary

This work addresses the challenge of out-of-distribution (OOD) generalization in periodic tasks for Transformer models. The authors propose a unified framework grounded in abstract algebra to formally characterize both simple and composite periodic structures, and introduce Coper—a controllable generation benchmark featuring two OOD settings: Hollow and Extrapolation. Systematic evaluation reveals that while Transformers can effectively memorize training-period patterns, they fail to generalize to unseen composite periodic configurations, exposing a fundamental limitation in their capacity for periodic reasoning. This study offers a novel perspective on the symbolic reasoning capabilities of Transformers and establishes both a theoretical foundation and a benchmark to guide future architectural innovations.

Technology Category

• Knowledge Representation and Reasoning: Computational Complexity of Reasoning
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning
• Reasoning under Uncertainty: Relational Probabilistic Models

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications

Abstract

Large language models (LLMs) based on the Transformer have demonstrated strong performance across diverse tasks. However, current models still exhibit substantial limitations in out-of-distribution (OOD) generalization compared with humans. We investigate this gap through periodicity, one of the basic OOD scenarios. Periodicity captures invariance amid variation. Periodicity generalization represents a model's ability to extract periodic patterns from training data and generalize to OOD scenarios. We introduce a unified interpretation of periodicity from the perspective of abstract algebra and reasoning, including both single and composite periodicity, to explain why Transformers struggle to generalize periodicity. Then we construct Coper about composite periodicity, a controllable generative benchmark with two OOD settings, Hollow and Extrapolation. Experiments reveal that periodicity generalization in Transformers is limited, where models can memorize periodic data during training, but cannot generalize to unseen composite periodicity. We release the source code to support future research.

Problem

• periodicity generalization
• out-of-distribution
• Transformer
• composite periodicity
• OOD generalization

Innovation

• periodicity generalization
• Transformer
• out-of-distribution
• composite periodicity
• abstract algebra