Selective Rotary Position Embedding

2025-11-21

AI Summary

This work addresses the lack of input adaptivity in positional encoding for language models. We propose Selective RoPE, an input-dependent dynamic rotary positional encoding that generalizes RoPE to arbitrary-angle rotations, maintaining compatibility with both softmax and linear attention mechanisms. Notably, it is the first method to explicitly uncover and exploit the implicit rotational structure inherent in softmax attention. Leveraging complex-valued representations, gating mechanisms, and selective rotation transformations, Selective RoPE enables implicit, dynamic modeling of positional information. It provides a unified interpretation of positional modeling and forgetting mechanisms in state space models and gated linear Transformers. Experiments demonstrate significant improvements over baselines on language modeling, long-sequence copying, state tracking, and retrieval tasks, validating its effectiveness and strong generalization capability across diverse sequence modeling challenges.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Natural Language Processing: (Large) Language Models
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Position information is essential for language modeling. In softmax transformers, Rotary Position Embeddings ( extit{RoPE}) encode positions through extit{fixed-angle} rotations, while in linear transformers, order is handled via input-dependent (selective) gating that decays past key-value associations. Selectivity has generally been shown to improve language-related tasks. Inspired by this, we introduce extit{Selective RoPE}, an extit{input-dependent} rotary embedding mechanism, that generalizes extit{RoPE}, and enables rotation in extit{arbitrary angles} for both linear and softmax transformers. We show that softmax attention already performs a hidden form of these rotations on query-key pairs, uncovering an implicit positional structure. We further show that in state-space models and gated linear transformers, the real part manages forgetting while the imaginary part encodes positions through rotations. We validate our method by equipping gated transformers with extit{Selective RoPE}, demonstrating that its input-dependent rotations improve performance in language modeling and on difficult sequence tasks like copying, state tracking, and retrieval.

Problem

• Developing input-dependent rotary embeddings for transformers
• Enabling arbitrary rotation angles in position encoding
• Improving language modeling and sequence task performance

Innovation

• Input-dependent rotary embedding mechanism for transformers
• Enables arbitrary rotation angles in both linear and softmax transformers
• Improves language modeling and sequence tasks performance