🤖 AI Summary
This study investigates the impact of fusion mechanisms between positional encoding and token embeddings on Transformer performance in long-sequence tasks. By systematically comparing additive, concatenation-with-projection, and scalar-gated strategies within a unified architecture—and further introducing a lightweight convolutional gating module to inject local inductive bias—the work demonstrates that learnable fusion approaches significantly outperform conventional additive fusion. Comprehensive evaluations, including controlled ablation studies, paired random-seed analyses, cross-dataset validation, and experiments across multiple positional encoding families, consistently reveal that the choice of fusion mechanism is a critical design factor for long-range modeling. The proposed method yields stable performance gains and exhibits strong generalization, particularly on long-text classification benchmarks such as ArXiv.
📝 Abstract
Transformers require positional encodings to represent sequence order, yet most prior work focuses on designing new positional encodings rather than examining how positional information is fused with token embeddings. In this paper, we study whether the fusion mechanism itself affects performance, particularly in long-sequence settings. We conduct a controlled empirical study comparing three canonical fusion strategies--element-wise addition, concatenation with projection, and scalar gated fusion--under identical Transformer architectures, data splits, and random seeds. Experiments on three text classification datasets spanning short (AG News), medium (IMDB), and long (ArXiv) sequences show that fusion choice has negligible impact on short texts but produces consistent gains on long documents. To verify that these gains are structural rather than stochastic, we perform paired-seed analysis and cross-dataset comparison across sequence-length regimes. Additional experiments on the ArXiv dataset indicate that the benefit of learnable fusion generalizes across multiple positional encoding families. Finally, we explore a lightweight convolutional gating mechanism that introduces local inductive bias at the fusion level, evaluated on long documents only. Our results indicate that positional-encoding fusion is a non-trivial design choice for long-sequence Transformers and should be treated as an explicit modeling decision rather than a fixed default.