ANI: Adaptive Numerical Injection for Unifying Semantic and Arithmetic Representations in Numerical Reasoning

2026-09-30

AI Summary

This study addresses the limitation that tokenization mechanisms in large language models fragment numbers, thereby impeding precise arithmetic reasoning, while existing numerical embedding methods neglect the semantic roles of numerals. To bridge this gap, this work proposes ANI, a hybrid framework that achieves the first context-aware selective injection of numerical embeddings. By integrating FoNE numerical embeddings with a context-aware gating mechanism, ANI adaptively injects numerical features into the model, effectively unifying token semantics with operational precision representations. Experimental results demonstrate that ANI yields a 9.5-point performance improvement on the MATH benchmark while maintaining robust performance across general language tasks.

Abstract

Precise numerical reasoning with Large Language Models (LLMs) is essential for expanding their applicability to complex real-world tasks. However, text-based tokenization often fragments numbers, significantly hindering precise arithmetic reasoning. Meanwhile, numerical embeddings, despite arithmetic precision, rely on context-agnostic substitution that disregards the semantic role of numbers as identifiers. To combine the complementary strengths, we propose \textbf{ANI (Adaptive Numerical Injection)}, a hybrid framework that governs the selective injection of numerical features based on the semantic context. By employing a context-aware gating mechanism, we selectively inject numerical embeddings (specifically FoNE) into the latent space, explicitly preserving nominal identifiers while enhancing quantitative operands. Through extensive evaluations across various LLMs, we demonstrate that ANI enhances MATH performance by 9.5 points over the official reference model, while maintaining robust performance on general linguistic benchmarks.

Problem

• Numerical Reasoning
• Large Language Models
• Tokenization
• Numerical Embeddings
• Semantic Representation

Innovation

• Adaptive Numerical Injection
• Context-Aware Gating Mechanism
• Numerical Reasoning
• Semantic-Arithmetic Unification
• Large Language Models