The Shape of Addition: Geometric Structures of Arithmetic in Large Language Models

2026-05-29

AI Summary

Large language models frequently err in basic arithmetic tasks due to a disconnect between their internal continuous representations and discrete outputs. This work addresses this issue by analyzing the geometric structure of residual streams in multi-operand addition, revealing and formally characterizing the Iso-Raw-Sum trajectory (IRST)—a latent pathway modulated by semantic number anchoring and continuous carry propagation. The authors propose a geometric slippage mechanism, wherein arithmetic errors arise from neural noise-induced deviations along this trajectory. To mitigate such errors, they introduce a lightweight probing method that disentangles coexisting latent signals from a single activation vector. Integrating a noise quantification model with geometric consistency verification, the approach substantially improves arithmetic reasoning accuracy and effectively detects and corrects quantization failures.

Abstract

Large Language Models exhibit paradoxical fragility in fundamental arithmetic, implying a disconnect between internal computation and discrete output. By analyzing the residual stream geometry during multi-operand addition, we identify the Iso-Raw-Sum Trajectory (IRST), a geometric structure where representations are anchored by semantic digits and modulated by continuous carry fibers. We propose the Noisy Quantization Model to explain this geometry, framing arithmetic errors as Geometric Slippages caused by internal neural noise pushing a continuous, latent Carry Potential across quantization thresholds. This geometric framework further elucidates Probe Versatility, explaining how lightweight probes can disentangle coexisting latent signals (such as ground truth versus hallucination) from a single activation vector. Finally, we validate these insights through a geometric consistency check method that effectively detects and corrects these quantization failures during inference. Our code is available at https://github.com/RL-MIND/Shape-of-Addition.

Problem

• arithmetic fragility
• large language models
• geometric structures
• quantization errors
• carry propagation

Innovation

• Iso-Raw-Sum Trajectory
• Geometric Slippage
• Noisy Quantization Model
• Carry Potential
• Probe Versatility