Analytic-Walk Rotary Positional Encodings for Graphs

2026-09-26

AI Summary

This study addresses the limitation of existing graph rotary position encodings in distinguishing different paths connecting identical endpoints by proposing AW-RoPE. This method pioneers placing rotary encodings on edges and accumulating features along all walks, enabling contributions from distinct paths to reinforce or cancel each other, thereby effectively identifying path differences defined solely by topological connectivity. For implementation, a differentiable linear solver is employed for the exact variant, while a finite-depth truncation is used for the sparse variant, both integrated with Performer kernels and a GIN architecture. Experimental results demonstrate that AW-RoPE reduces nRMSE by 15%–58% on synthetic tasks and achieves state-of-the-art performance across multiple real-world benchmarks, significantly outperforming existing baselines.

Abstract

Rotary position encodings make attention sensitive to relative position, but extending them to graphs requires choosing how graph structure enters the rotation. Previous works assign each node a rotation from spectral coordinates, so the rotary factor between two nodes depends only on their endpoints and cannot distinguish the routes connecting them. We introduce \textit{Analytic-Walk Rotary Positional Encodings} (AW-RoPE), which place the rotations on edges and sum the transported features over all walks, so contributions along different routes can reinforce or cancel. An exact variant evaluates the complete sum by a differentiable linear solve, and a sparse variant truncates it at a finite depth. We prove forward and parameter-derivative truncation bounds at fixed inputs and parameters. Both variants act on projected queries and keys, and the sparse recurrence also augments message-passing networks. Across five synthetic tasks both variants reduce nRMSE by $15$--$58\%$ relative to the strongest baseline, and on real superpixel, peptide and OGB benchmarks the sparse recurrence attains the best mean on every dataset with Performer kernels and on twelve of thirteen datasets with GIN. Analysis shows that AW-RoPE can distinguish routes whose only cue is how two endpoints are connected, while node-wise rotary encodings cannot.

Problem

• Rotary Positional Encodings
• Graph Structure
• Path Distinction
• Spectral Coordinates

Innovation

• Rotary Positional Encoding
• Graph Neural Networks
• Analytic Walk
• Message Passing
• Differentiable Linear Solve