Addition in Four Movements: Mapping Layer-wise Information Trajectories in LLMs

2025-06-09

AI Summary

This work investigates the internal computational mechanisms underlying multi-digit addition in LLaMA-3-8B-Instruct. Addressing the question of *how models perform arithmetic reasoning step-by-step*, we propose a layer-wise activation analysis method combining linear probing with logit lens. For the first time, we systematically uncover a four-stage, bottom-up information evolution pathway during forward propagation: (1) formulaic structure representation, (2) core computational feature extraction, (3) numerical abstraction, and (4) answer generation. This pathway is highly interpretable and confirms that the model relies on hierarchical internal computation rather than memorized pattern matching. Experiments demonstrate near-perfect inter-layer decodability (>99%) of digit-wise intermediate results, and the correct answer token consistently ranks first at the output layer. All code and data are publicly released.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Multi-digit addition is a clear probe of the computational power of large language models. To dissect the internal arithmetic processes in LLaMA-3-8B-Instruct, we combine linear probing with logit-lens inspection. Inspired by the step-by-step manner in which humans perform addition, we propose and analyze a coherent four-stage trajectory in the forward pass:Formula-structure representations become linearly decodable first, while the answer token is still far down the candidate list.Core computational features then emerge prominently.At deeper activation layers, numerical abstractions of the result become clearer, enabling near-perfect detection and decoding of the individual digits in the sum.Near the output, the model organizes and generates the final content, with the correct token reliably occupying the top rank.This trajectory suggests a hierarchical process that favors internal computation over rote memorization. We release our code and data to facilitate reproducibility.

Problem

• Probing multi-digit addition to assess LLM computational power
• Analyzing four-stage arithmetic process in LLaMA-3-8B-Instruct
• Mapping hierarchical information trajectory during addition tasks

Innovation

• Combines linear probing with logit-lens inspection
• Proposes a four-stage trajectory for addition
• Detects numerical abstractions in deeper layers