Frontier Language Models Struggle to Copy: Text Can Be Better Viewed in 2D

2026-07-17

AI Summary

Large language models struggle to accurately reproduce input sequences that are far shorter than their context window. This work addresses this limitation by introducing a novel two-dimensional formulation of text as a grid and proposing two-dimensional Rotary Position Embedding (2D-RoPE), which explicitly incorporates row- and column-wise positional information into the Transformer architecture. The proposed method dramatically simplifies long-range exact copying tasks, enabling even shallow models to achieve perfect generalization on inputs hundreds of times longer than those seen during training. Experiments demonstrate that a 1.4B-parameter model equipped with 2D-RoPE consistently outperforms standard position encodings across both synthetic benchmarks and large-scale DCLM pretraining, confirming the effectiveness and scalability of the approach.

Abstract

While large language models (LLMs) can solve advanced reasoning problems in seconds, we show that even frontier models fail to perform a much simpler operation: exactly copying an input string that lies well within their context windows. We attribute this failure to positional encodings in Transformer architectures, whose inductive bias favors copying through a shortcut based on matching local contexts rather than carefully locating the corresponding input positions. To address this issue, we introduce 2D-RoPE, which organizes text into a 2D grid rather than a 1D sequence and assigns each token a row ID and a column ID. Under this view, copying becomes simply retrieving input tokens at a fixed column offset, which makes the task easy to learn. In synthetic copy experiments, shallow Transformers with 2D-RoPE achieve perfect copying at input lengths hundreds of times longer than those seen during training, whereas standard positional encodings fall far behind. We further show that the advantage of 2D-RoPE language models on copy tasks consistently holds in large-scale pretraining on DCLM with model sizes up to 1.4B parameters. Overall, our results suggest that viewing text in 2D can benefit language modeling, and we hope this encourages future work to further explore the potential of 2D positional encodings.

Problem

• copying
• positional encodings
• Transformer
• inductive bias
• language models

Innovation

• 2D-RoPE
• positional encoding
• copy task
• Transformer architecture
• inductive bias