WavePrune: One period is often enough for RoPE

2026-10-03

AI Summary

This study addresses the limitation of Rotary Position Embedding (RoPE), where its inherent periodicity induces positional aliasing and constrains long-context modeling capabilities. To overcome this, we propose WavePrune, which identifies structural redundancy beyond the first rotation period. By employing a truncation strategy that restricts each channel to retain only its initial period, our method effectively eliminates attention interference. Furthermore, WavePrune integrates fine-grained sparsity with hardware-aligned CUDA kernels and FlashAttention-2 acceleration to enable tuning-free inference. This work achieves simultaneous improvements in both performance and efficiency, demonstrating significantly higher HELMET scores alongside reduced validation loss. Additionally, it yields 1.15× and 1.24× speedups during the prefill and decoding stages, respectively.

Abstract

Rotary Position Embedding (RoPE) encodes token positions by rotating each two-dimensional channel of the query and key vectors at a channel-specific frequency, making the attention logits invariant to a common shift of positions. However, this rotation is periodic, and it leads to position aliasing where relative positions separated by a full rotation period become hard to tell apart. To address this, we propose WavePrune, which restricts each channel to its first rotation period. We show that it removes the distractions in attention maps created by position aliasing and improves overall long-context performance. Specifically, WavePrune raises the HELMET score on four of five models we test without any extra tuning (e.g., 35.7 -> 40.0 on Qwen3-8B). When pretraining models from scratch, WavePrune also achieves lower validation loss at extrapolated lengths than pretraining without it. Because WavePrune restricts each channel to a sliding window, it induces a fine-grained sparsity that our hardware-aligned CUDA kernels exploit for 1.15x prefill and 1.24x decoding speedups over FlashAttention-2 at 32K context. Together, these results show that RoPE's periodic structure, widely regarded as essential, is largely redundant beyond the first rotation period.

Problem

• Rotary Position Embedding
• position aliasing
• long-context performance
• periodicity

Innovation

• WavePrune
• Rotary Position Embedding
• Position Aliasing
• Long-context
• Fine-grained Sparsity