Beyond Sequence Order: Syntax-Informed Positional Embeddings for Transformers

2026-08-06

AI Summary

This work addresses the inability of conventional Transformer positional encodings to model hierarchical syntactic structures in language. The authors propose SiPE, a lightweight syntax-aware positional encoding method that learns syntactic priors from dependency parse trees during pretraining and injects them into mainstream positional encodings—absolute, relative, and rotary—via multiplicative coupling and additive fusion, compatible with both encoder and decoder architectures. For the first time, the study systematically investigates optimal strategies for integrating syntactic priors across different architectural settings. Without introducing additional inference overhead, SiPE yields substantial performance gains: a 10.3% absolute improvement in SyntaxGym accuracy, a 9.0% reduction in perplexity, and up to an 8.2% gain on GLUE benchmarks, effectively overcoming the longstanding trade-off between syntactic generalization and language modeling performance.

Abstract

Positional embeddings (PE) in Transformers encode token distance and order but are largely agnostic to \textit{syntactic structure}. We introduce \textbf{S}yntax-\textbf{i}nformed \textbf{P}ositional \textbf{E}mbeddings (\textbf{SiPE}), which learns a lightweight syntactic prior from dependency parses during pretraining and injects it across all three dominant PE families (absolute, relative, rotary), for both encoders and decoders, leaving self-attention and the rest of the architecture untouched. We isolate \emph{where} and \emph{how} the prior should enter the model, and find it depends on the architecture: for autoregressive decoders that use relative PE, the prior is strongest when coupled multiplicatively with the relative-position term of the attention score, outperforming injection into the input embeddings, into self-attention, or into the positional and attention terms jointly---while for encoders it is best added directly to the input embeddings, composing with each encoder's native positional mechanism. We find that models pre-trained with SiPE improve on the SyntaxGym benchmark by up to $10.3\%$ while simultaneously reducing perplexity by $9.0\%$ over a base model with no syntactic supervision---a metric nearly every existing syntax-injection method instead degrades. Crucially, these gains extend beyond syntactic generalization: SiPE also improves real-world language understanding, raising scores on the GLUE benchmark by up to $8.2\%$ over a model trained without it. Unlike existing syntactic language models that marginalize over many parses at inference or discard syntax at runtime, SiPE conditions on a single parse, establishing a new Pareto frontier between syntactic supervision and inference cost.

Problem

• positional embeddings
• syntactic structure
• Transformers
• syntax-awareness
• language modeling

Innovation

• Syntax-aware Positional Embeddings
• Dependency Parsing
• Transformer Architecture
• Syntactic Generalization
• Efficient Syntax Integration