🤖 AI Summary
Existing positional encodings rely on predefined sequential orders, limiting their ability to model non-sequential yet causally related features. To address this, we propose Causal-Aware Positional Encoding (CAPE): first, a generalized structural equation model learns a causal directed acyclic graph (DAG); second, the DAG is embedded in hyperbolic space to preserve both causal strength and specificity; third, a rotation-based transformation seamlessly integrates the encoding into Transformer self-attention. We theoretically prove that CAPE satisfies causal distance metric properties and exhibits generalization decay—key desiderata for causal representation learning—while remaining robust to positional perturbations. Empirical evaluations on synthetic and real-world datasets demonstrate that CAPE significantly enhances Transformer’s representational capacity for non-sequential causal data. The implementation is publicly available.
📝 Abstract
Positional encoding is essential for supplementing transformer with positional information of tokens. Existing positional encoding methods demand predefined token/feature order, rendering them unsuitable for real-world data with non-sequential yet causally-related features. To address this limitation, we propose CAPE, a novel method that identifies underlying causal structure over non-sequential features as a weighted directed acyclic graph (DAG) using generalized structural equation modeling. The DAG is then embedded in hyperbolic space where its geometric structure is well-preserved using a hyperboloid model-based approach that effectively captures two important causal graph properties (causal strength & causal specificity). This step yields causality-aware positional encodings for the features, which are converted into their rotary form for integrating with transformer's self-attention mechanism. Theoretical analysis reveals that CAPE-generated rotary positional encodings possess three valuable properties for enhanced self-attention, including causal distance-induced attenuation, causal generality-induced attenuation, and robustness to positional disturbances. We evaluate CAPE over both synthetic and real-word datasets, empirically demonstrating its theoretical properties and effectiveness in enhancing transformer for data with non-sequential features. Our code is available at https://github.com/Catchxu/CAPE.