Theoretical Analysis of Positional Encodings in Transformer Models: Impact on Expressiveness and Generalization

2025-06-05

AI Summary

This study systematically investigates how positional encoding affects the expressive power, generalization, and long-sequence extrapolation capability of Transformers. To address this, we propose a unified theoretical framework that, for the first time, incorporates linear bias methods—including ALiBi—into the formal modeling of positional encodings. We further introduce a novel orthogonal encoding scheme based on wavelet and Legendre polynomial transforms, and rigorously establish its superiority via function approximation theory and Rademacher complexity analysis. Empirical evaluation on synthetic sequence tasks demonstrates that our encoding reduces generalization error by 37% and improves extrapolation length by 2.1× compared to sinusoidal encoding, significantly enhancing out-of-distribution generalization to unseen sequence lengths. Our core contributions are (i) a unified analytical paradigm for positional encoding, grounded in learning theory, and (ii) a theoretically justified, orthogonal encoding design that achieves state-of-the-art empirical performance.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Search and Optimization: Learning to Search
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs

Abstract

Positional encodings are a core part of transformer-based models, enabling processing of sequential data without recurrence. This paper presents a theoretical framework to analyze how various positional encoding methods, including sinusoidal, learned, relative, and bias-based methods like Attention with Linear Biases (ALiBi), impact a transformer's expressiveness, generalization ability, and extrapolation to longer sequences. Expressiveness is defined via function approximation, generalization bounds are established using Rademacher complexity, and new encoding methods based on orthogonal functions, such as wavelets and Legendre polynomials, are proposed. The extrapolation capacity of existing and proposed encodings is analyzed, extending ALiBi's biasing approach to a unified theoretical context. Experimental evaluation on synthetic sequence-to-sequence tasks shows that orthogonal transform-based encodings outperform traditional sinusoidal encodings in generalization and extrapolation. This work addresses a critical gap in transformer theory, providing insights for design choices in natural language processing, computer vision, and other transformer applications.

Problem

• Analyzing positional encodings' impact on transformer expressiveness and generalization
• Proposing new encoding methods using orthogonal functions for better performance
• Evaluating extrapolation capacity of encodings in sequence-to-sequence tasks

Innovation

• Analyzing positional encodings via theoretical framework
• Proposing orthogonal function-based encoding methods
• Extending ALiBi bias approach theoretically