Testing LLM Arithmetic Reasoning Generalization with Automatic Numeric-Remapping Attacks

2026-06-02

AI Summary

This work demonstrates that large language models (LLMs) exhibit high sensitivity to minor numerical perturbations in natural language arithmetic reasoning, where even slight digit substitutions can substantially degrade performance. To systematically exploit this vulnerability, the authors propose the first fully automated and scalable numerical remapping attack framework. The approach extracts symbolic expressions from input problems, generates constraint-preserving numerical replacements, employs LLM-guided deterministic editing to rewrite questions, and automatically recomputes ground-truth answers. A staged validation pipeline coupled with a high-confidence auditing mechanism ensures that perturbations remain effective while preserving the original reasoning structure. Experiments reveal that the method reduces accuracy on GSM8K by 12.16–25.82 percentage points, whereas datasets like MAWPS and MultiArith maintain over 98% accuracy, highlighting the critical role of dataset structure in numerical robustness.

Abstract

Large language models achieve strong performance on arithmetic reasoning benchmarks, and one common response to arithmetic brittleness is to delegate computation to code. Yet models are still often used in settings where they must reason directly from natural language, and trustworthy models should solve small-number arithmetic word problems without external tools. Prior work shows that LLMs are sensitive to numerical variation: a model may solve an original problem but fail on structurally similar variants requiring the same reasoning procedure with different numbers. We ask whether this fragility persists under a stricter setting involving small, schema-preserving numeric changes that retain the original reasoning program and avoid large-number stress tests. We introduce an automatic algorithm for generating numeric-remapping attacks on arithmetic word problems. Unlike template-based perturbation methods requiring manual schemas or constraints, our approach derives problem-specific symbolic representations, generates constrained numeric remappings, recomputes gold answers, and realizes transformed questions through deterministic edits guided by LLM-generated edit plans. Stage-wise validation and a high-confidence audit retain reliable attacks, making the pipeline scalable with limited human intervention. We evaluate DeepSeek-R1 (70B), Gemma4 (31B), and GPT-OSS (120B) on GSM8K, MAWPS, and MultiArith. On GSM8K, completed runs show conditional accuracy drops of 12.16 to 25.82 percentage points. MAWPS and MultiArith are far more stable, with most attacked accuracies near or above 98%. These results show that numeric-remapping robustness depends strongly on dataset structure: GSM8K remains sensitive even when reasoning programs are preserved and answers are recomputed, while shorter, more regular datasets are more robust.

Problem

• arithmetic reasoning
• numeric remapping
• generalization
• large language models
• robustness

Innovation

• numeric-remapping attacks
• arithmetic reasoning
• symbolic representation
• LLM robustness
• automatic perturbation